2023-07-29T16:48:01.949 INFO:root:teuthology version: 0.0.1.dev172+g6114016 2023-07-29T16:48:01.950 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} --archive /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/orig.config.yaml 2023-07-29T16:48:01.980 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-29T16:48:02.062 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902 branch: reef-release description: orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} email: null first_in_suite: false job_id: '7356902' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef-release ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 workunit: branch: reef-release sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 2786 sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi129.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - cephadm: allow_ptrace: false avoid_pacific_features: true cephadm_branch: v16.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v16.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 61140164d2579c30ffff3970615e0f82e40c6470 timestamp: 2023-07-29_14:03:42 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-29T16:48:02.063 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa; will attempt to use it 2023-07-29T16:48:02.063 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks 2023-07-29T16:48:02.064 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-29T16:48:02.065 INFO:teuthology.task.internal:Checking packages... 2023-07-29T16:48:02.095 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '0484097b0572e96f42cf6402cbe0ac8dcb046577' 2023-07-29T16:48:02.095 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-29T16:48:02.096 INFO:teuthology.packaging:ref: None 2023-07-29T16:48:02.096 INFO:teuthology.packaging:tag: None 2023-07-29T16:48:02.096 INFO:teuthology.packaging:branch: reef-release 2023-07-29T16:48:02.096 INFO:teuthology.packaging:sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:48:02.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef-release 2023-07-29T16:48:02.472 INFO:teuthology.task.internal:Found packages for ceph version 18.1.3-11-g0484097b-1focal 2023-07-29T16:48:02.472 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-29T16:48:02.480 INFO:teuthology.task.internal:no buildpackages task found 2023-07-29T16:48:02.480 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-29T16:48:02.520 INFO:teuthology.task.internal:Saving configuration 2023-07-29T16:48:02.532 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-29T16:48:02.545 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T16:48:02.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 16:42:02.580587', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-29T16:48:02.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 16:42:02.582212', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-29T16:48:02.598 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-29T16:48:02.605 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2023-07-29T16:48:02.605 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2023-07-29T16:48:02.605 INFO:teuthology.run_tasks:Running task console_log... 2023-07-29T16:48:02.728 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6788b52160>, signals=[15]) 2023-07-29T16:48:02.730 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-29T16:48:02.736 INFO:teuthology.task.internal:Opening connections... 2023-07-29T16:48:02.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2023-07-29T16:48:02.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:48:02.788 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2023-07-29T16:48:02.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:48:02.839 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-29T16:48:02.846 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2023-07-29T16:48:02.872 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2023-07-29T16:48:02.872 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2023-07-29T16:48:02.921 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2023-07-29T16:48:02.921 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-29T16:48:02.921 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2023-07-29T16:48:02.921 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2023-07-29T16:48:02.921 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="20.04" 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=focal 2023-07-29T16:48:02.922 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=focal 2023-07-29T16:48:02.923 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2023-07-29T16:48:02.948 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2023-07-29T16:48:02.955 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2023-07-29T16:48:02.955 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2023-07-29T16:48:02.961 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Ubuntu" 2023-07-29T16:48:02.961 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-29T16:48:02.961 INFO:teuthology.orchestra.run.smithi129.stdout:ID=ubuntu 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE=debian 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="20.04" 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-29T16:48:02.962 INFO:teuthology.orchestra.run.smithi129.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-29T16:48:02.963 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_CODENAME=focal 2023-07-29T16:48:02.963 INFO:teuthology.orchestra.run.smithi129.stdout:UBUNTU_CODENAME=focal 2023-07-29T16:48:02.963 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-07-29T16:48:02.989 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-29T16:48:02.998 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-29T16:48:03.020 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-29T16:48:03.021 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2023-07-29T16:48:03.023 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2023-07-29T16:48:03.028 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-29T16:48:03.045 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-29T16:48:03.045 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2023-07-29T16:48:03.068 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2023-07-29T16:48:03.080 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-29T16:48:03.144 INFO:teuthology.run_tasks:Running task kernel... 2023-07-29T16:48:03.159 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-29T16:48:03.160 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-29T16:48:03.160 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-29T16:48:03.161 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-29T16:48:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:48:03.167 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-07-29T16:48:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:48:03.168 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2023-07-29T16:48:03.174 INFO:teuthology.orchestra.run.smithi078.stdout:5.4.0-124-generic 2023-07-29T16:48:03.175 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.4.0-124-generic 2023-07-29T16:48:03.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2023-07-29T16:48:03.177 INFO:teuthology.orchestra.run.smithi129.stdout:5.4.0-124-generic 2023-07-29T16:48:03.177 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.4.0-124-generic 2023-07-29T16:48:03.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2023-07-29T16:48:03.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2023-07-29T16:48:03.320 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2023-07-29T16:48:03.588 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-29T16:48:03.591 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-07-29T16:48:03.595 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-29T16:48:03.595 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-07-29T16:48:03.675 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-07-29T16:48:03.676 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-07-29T16:48:03.679 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-07-29T16:48:03.679 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-07-29T16:48:03.991 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [857 kB] 2023-07-29T16:48:03.991 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,721 kB] 2023-07-29T16:48:04.108 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,721 kB] 2023-07-29T16:48:04.166 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [857 kB] 2023-07-29T16:48:04.175 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [453 kB] 2023-07-29T16:48:04.183 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [453 kB] 2023-07-29T16:48:04.185 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.9 kB] 2023-07-29T16:48:04.186 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,143 kB] 2023-07-29T16:48:04.191 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.9 kB] 2023-07-29T16:48:04.191 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,143 kB] 2023-07-29T16:48:04.234 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [33.3 kB] 2023-07-29T16:48:04.234 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [300 kB] 2023-07-29T16:48:04.239 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [33.3 kB] 2023-07-29T16:48:04.240 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [300 kB] 2023-07-29T16:48:04.240 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [576 B] 2023-07-29T16:48:04.241 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [739 kB] 2023-07-29T16:48:04.295 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [576 B] 2023-07-29T16:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,092 kB] 2023-07-29T16:48:04.300 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,092 kB] 2023-07-29T16:48:04.307 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [739 kB] 2023-07-29T16:48:04.312 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [626 kB] 2023-07-29T16:48:04.313 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [260 kB] 2023-07-29T16:48:04.314 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [260 kB] 2023-07-29T16:48:04.315 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.3 kB] 2023-07-29T16:48:04.316 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-07-29T16:48:04.316 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-07-29T16:48:04.316 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.3 kB] 2023-07-29T16:48:04.317 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-07-29T16:48:04.317 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-07-29T16:48:04.317 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-07-29T16:48:04.318 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-07-29T16:48:04.318 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-07-29T16:48:04.318 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-07-29T16:48:04.318 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-07-29T16:48:04.318 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-07-29T16:48:04.326 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [626 kB] 2023-07-29T16:48:04.326 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-07-29T16:48:04.327 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-07-29T16:48:04.350 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-07-29T16:48:04.350 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-07-29T16:48:04.358 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-07-29T16:48:04.358 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-07-29T16:48:04.359 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-07-29T16:48:04.359 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-07-29T16:48:04.380 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-07-29T16:48:04.381 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-07-29T16:48:04.669 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,338 kB] 2023-07-29T16:48:04.679 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,338 kB] 2023-07-29T16:48:04.868 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [370 kB] 2023-07-29T16:48:04.879 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [370 kB] 2023-07-29T16:48:04.883 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.0 kB] 2023-07-29T16:48:04.884 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,031 kB] 2023-07-29T16:48:04.895 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.0 kB] 2023-07-29T16:48:04.896 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,031 kB] 2023-07-29T16:48:04.954 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [32.0 kB] 2023-07-29T16:48:04.955 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [285 kB] 2023-07-29T16:48:04.961 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [576 B] 2023-07-29T16:48:04.961 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [866 kB] 2023-07-29T16:48:04.965 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [32.0 kB] 2023-07-29T16:48:04.966 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [285 kB] 2023-07-29T16:48:04.972 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [576 B] 2023-07-29T16:48:04.973 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [607 kB] 2023-07-29T16:48:04.979 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [607 kB] 2023-07-29T16:48:04.986 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [866 kB] 2023-07-29T16:48:05.195 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [180 kB] 2023-07-29T16:48:05.202 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [180 kB] 2023-07-29T16:48:05.691 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [18.8 kB] 2023-07-29T16:48:05.699 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-07-29T16:48:05.702 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-07-29T16:48:05.707 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [18.8 kB] 2023-07-29T16:48:05.713 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-07-29T16:48:05.715 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-07-29T16:48:05.716 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-07-29T16:48:05.718 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-07-29T16:48:05.729 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-07-29T16:48:05.731 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-07-29T16:48:07.863 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 16.5 MB in 3s (6,487 kB/s) 2023-07-29T16:48:08.119 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 16.5 MB in 3s (6,319 kB/s) 2023-07-29T16:48:09.219 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2023-07-29T16:48:09.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-29T16:48:09.315 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2023-07-29T16:48:09.511 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2023-07-29T16:48:09.511 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2023-07-29T16:48:09.607 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2023-07-29T16:48:09.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-29T16:48:09.687 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2023-07-29T16:48:09.687 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-5.4.0-155 linux-headers-5.4.0-155-generic 2023-07-29T16:48:09.687 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-generic linux-image-5.4.0-155-generic 2023-07-29T16:48:09.688 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-5.4.0-155-generic linux-modules-extra-5.4.0-155-generic 2023-07-29T16:48:09.688 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2023-07-29T16:48:09.688 INFO:teuthology.orchestra.run.smithi129.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-07-29T16:48:09.688 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2023-07-29T16:48:09.689 INFO:teuthology.orchestra.run.smithi129.stdout: thermald 2023-07-29T16:48:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2023-07-29T16:48:09.748 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2023-07-29T16:48:09.749 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-5.4.0-155 linux-headers-5.4.0-155-generic 2023-07-29T16:48:09.749 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.4.0-155-generic linux-modules-5.4.0-155-generic 2023-07-29T16:48:09.749 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-extra-5.4.0-155-generic 2023-07-29T16:48:09.750 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2023-07-29T16:48:09.750 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-generic linux-image-generic 2023-07-29T16:48:09.868 INFO:teuthology.orchestra.run.smithi129.stdout:3 upgraded, 5 newly installed, 0 to remove and 226 not upgraded. 2023-07-29T16:48:09.868 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 77.0 MB of archives. 2023-07-29T16:48:09.868 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 380 MB of additional disk space will be used. 2023-07-29T16:48:09.868 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-155-generic amd64 5.4.0-155.172 [15.0 MB] 2023-07-29T16:48:09.920 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2023-07-29T16:48:09.920 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2023-07-29T16:48:10.102 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2023-07-29T16:48:10.103 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-5.4.0-155 linux-headers-5.4.0-155-generic 2023-07-29T16:48:10.103 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-generic linux-image-5.4.0-155-generic 2023-07-29T16:48:10.103 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-5.4.0-155-generic linux-modules-extra-5.4.0-155-generic 2023-07-29T16:48:10.104 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2023-07-29T16:48:10.104 INFO:teuthology.orchestra.run.smithi078.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-07-29T16:48:10.104 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2023-07-29T16:48:10.104 INFO:teuthology.orchestra.run.smithi078.stdout: thermald 2023-07-29T16:48:10.175 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2023-07-29T16:48:10.175 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-5.4.0-155 linux-headers-5.4.0-155-generic 2023-07-29T16:48:10.175 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-155-generic linux-modules-5.4.0-155-generic 2023-07-29T16:48:10.175 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-extra-5.4.0-155-generic 2023-07-29T16:48:10.177 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2023-07-29T16:48:10.177 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-generic linux-image-generic 2023-07-29T16:48:10.437 INFO:teuthology.orchestra.run.smithi078.stdout:3 upgraded, 5 newly installed, 0 to remove and 226 not upgraded. 2023-07-29T16:48:10.437 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 77.0 MB of archives. 2023-07-29T16:48:10.438 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 380 MB of additional disk space will be used. 2023-07-29T16:48:10.438 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-155-generic amd64 5.4.0-155.172 [15.0 MB] 2023-07-29T16:48:10.677 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-155-generic amd64 5.4.0-155.172 [10.5 MB] 2023-07-29T16:48:10.909 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-155-generic amd64 5.4.0-155.172 [39.2 MB] 2023-07-29T16:48:11.873 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.155.151 [1,900 B] 2023-07-29T16:48:11.873 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.155.151 [2,600 B] 2023-07-29T16:48:11.874 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-155 all 5.4.0-155.172 [11.0 MB] 2023-07-29T16:48:12.117 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-155-generic amd64 5.4.0-155.172 [1,341 kB] 2023-07-29T16:48:12.160 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.155.151 [2,464 B] 2023-07-29T16:48:12.212 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-155-generic amd64 5.4.0-155.172 [10.5 MB] 2023-07-29T16:48:12.670 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-155-generic amd64 5.4.0-155.172 [39.2 MB] 2023-07-29T16:48:12.811 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 77.0 MB in 2s (32.0 MB/s) 2023-07-29T16:48:13.003 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-5.4.0-155-generic. 2023-07-29T16:48:14.698 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-07-29T16:48:14.703 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../0-linux-modules-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:14.767 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:14.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.155.151 [1,900 B] 2023-07-29T16:48:14.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.155.151 [2,600 B] 2023-07-29T16:48:14.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-155 all 5.4.0-155.172 [11.0 MB] 2023-07-29T16:48:15.091 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-155-generic amd64 5.4.0-155.172 [1,341 kB] 2023-07-29T16:48:15.136 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.155.151 [2,464 B] 2023-07-29T16:48:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 77.0 MB in 5s (15.6 MB/s) 2023-07-29T16:48:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-5.4.0-155-generic. 2023-07-29T16:48:16.659 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-image-5.4.0-155-generic. 2023-07-29T16:48:16.672 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../1-linux-image-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:16.768 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:17.742 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-07-29T16:48:17.746 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../0-linux-modules-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:17.854 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-155-generic. 2023-07-29T16:48:17.861 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:17.867 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:17.912 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-extra-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:19.736 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-5.4.0-155-generic. 2023-07-29T16:48:19.753 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../1-linux-image-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:19.862 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-155-generic. 2023-07-29T16:48:21.020 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:21.080 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-extra-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:22.722 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../3-linux-generic_5.4.0.155.151_amd64.deb ... 2023-07-29T16:48:22.849 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-generic (5.4.0.155.151) over (5.4.0.124.125) ... 2023-07-29T16:48:23.192 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.155.151_amd64.deb ... 2023-07-29T16:48:23.319 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-generic (5.4.0.155.151) over (5.4.0.124.125) ... 2023-07-29T16:48:23.664 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.4.0-155. 2023-07-29T16:48:23.679 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../5-linux-headers-5.4.0-155_5.4.0-155.172_all.deb ... 2023-07-29T16:48:23.722 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.4.0-155 (5.4.0-155.172) ... 2023-07-29T16:48:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../3-linux-generic_5.4.0.155.151_amd64.deb ... 2023-07-29T16:48:26.150 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-generic (5.4.0.155.151) over (5.4.0.124.125) ... 2023-07-29T16:48:26.500 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.155.151_amd64.deb ... 2023-07-29T16:48:26.628 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-generic (5.4.0.155.151) over (5.4.0.124.125) ... 2023-07-29T16:48:26.982 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.4.0-155. 2023-07-29T16:48:26.996 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../5-linux-headers-5.4.0-155_5.4.0-155.172_all.deb ... 2023-07-29T16:48:27.056 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.4.0-155 (5.4.0-155.172) ... 2023-07-29T16:48:28.829 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.4.0-155-generic. 2023-07-29T16:48:28.851 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../6-linux-headers-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:31.511 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.155.151_amd64.deb ... 2023-07-29T16:48:31.638 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-generic (5.4.0.155.151) over (5.4.0.124.125) ... 2023-07-29T16:48:31.973 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.4.0-155 (5.4.0-155.172) ... 2023-07-29T16:48:32.101 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:32.529 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.4.0-155-generic. 2023-07-29T16:48:32.551 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../6-linux-headers-5.4.0-155-generic_5.4.0-155.172_amd64.deb ... 2023-07-29T16:48:32.595 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:33.319 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:34.453 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-155-generic 2023-07-29T16:48:34.453 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-155-generic 2023-07-29T16:48:34.621 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:34.731 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-generic (5.4.0.155.151) ... 2023-07-29T16:48:34.857 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-extra-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:35.885 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.155.151_amd64.deb ... 2023-07-29T16:48:36.022 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-generic (5.4.0.155.151) over (5.4.0.124.125) ... 2023-07-29T16:48:36.058 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-generic (5.4.0.155.151) ... 2023-07-29T16:48:36.184 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-generic (5.4.0.155.151) ... 2023-07-29T16:48:36.302 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for linux-image-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:36.371 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.4.0-155 (5.4.0-155.172) ... 2023-07-29T16:48:36.377 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-07-29T16:48:36.377 INFO:teuthology.orchestra.run.smithi129.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-155-generic 2023-07-29T16:48:36.519 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:37.746 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:38.903 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-155-generic 2023-07-29T16:48:38.904 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-155-generic 2023-07-29T16:48:39.066 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:39.191 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-generic (5.4.0.155.151) ... 2023-07-29T16:48:39.317 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-extra-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:41.969 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-generic (5.4.0.155.151) ... 2023-07-29T16:48:42.095 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-generic (5.4.0.155.151) ... 2023-07-29T16:48:42.239 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for linux-image-5.4.0-155-generic (5.4.0-155.172) ... 2023-07-29T16:48:42.319 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-07-29T16:48:42.320 INFO:teuthology.orchestra.run.smithi078.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-155-generic 2023-07-29T16:48:58.109 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-07-29T16:48:58.109 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub' 2023-07-29T16:48:58.117 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-29T16:48:58.143 INFO:teuthology.orchestra.run.smithi129.stdout:Generating grub configuration file ... 2023-07-29T16:48:58.461 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.4.0-155-generic 2023-07-29T16:48:58.484 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.4.0-155-generic 2023-07-29T16:48:58.856 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-29T16:48:58.867 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-29T16:49:01.014 INFO:teuthology.orchestra.run.smithi129.stdout:done 2023-07-29T16:49:01.428 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2023-07-29T16:49:01.472 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2023-07-29T16:49:01.472 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2023-07-29T16:49:01.472 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2023-07-29T16:49:01.473 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2023-07-29T16:49:01.473 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 19 2023-07-29T16:49:01.473 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2023-07-29T16:49:01.473 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2023-07-29T16:49:01.473 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2023-07-29T16:49:01.473 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.4.0.155.151 2023-07-29T16:49:01.474 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-29T16:49:01.474 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.4.0-155-generic, linux-modules-extra-5.4.0-155-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-29T16:49:01.474 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2023-07-29T16:49:01.474 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2023-07-29T16:49:01.474 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2023-07-29T16:49:01.474 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2023-07-29T16:49:01.475 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-155-generic 2023-07-29T16:49:01.475 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-29T16:49:01.475 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-29T16:49:01.475 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-07-29T16:49:01.476 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2023-07-29T16:49:01.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2023-07-29T16:49:01.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2023-07-29T16:49:01.704 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-29T16:49:01.717 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-07-29T16:49:01.742 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-07-29T16:49:01.829 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-07-29T16:49:03.417 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2023-07-29T16:49:03.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-29T16:49:03.509 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2023-07-29T16:49:03.706 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2023-07-29T16:49:03.707 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2023-07-29T16:49:03.920 INFO:teuthology.orchestra.run.smithi129.stdout:linux-image-generic is already the newest version (5.4.0.155.151). 2023-07-29T16:49:03.920 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 0 newly installed, 0 to remove and 226 not upgraded. 2023-07-29T16:49:03.922 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2023-07-29T16:49:03.942 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2023-07-29T16:49:03.943 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2023-07-29T16:49:03.943 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2023-07-29T16:49:03.943 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2023-07-29T16:49:03.943 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 19 2023-07-29T16:49:03.943 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2023-07-29T16:49:03.943 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.4.0.155.151 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.4.0-155-generic, linux-modules-extra-5.4.0-155-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2023-07-29T16:49:03.944 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2023-07-29T16:49:03.945 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2023-07-29T16:49:03.945 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-07-29T16:49:03.950 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.GVRoPkNZP9 2023-07-29T16:49:03.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub/grub.cfg /tmp/tmp.GVRoPkNZP9 2023-07-29T16:49:04.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.GVRoPkNZP9 2023-07-29T16:49:04.038 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.GVRoPkNZP9 is 9KB 2023-07-29T16:49:04.051 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.GVRoPkNZP9 2023-07-29T16:49:04.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-07-29T16:49:04.113 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:49:04.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-07-29T16:49:04.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-07-29T16:49:04.160 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-155-generic 2023-07-29T16:49:04.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo update-grub 2023-07-29T16:49:04.222 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-07-29T16:49:04.223 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub' 2023-07-29T16:49:04.235 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-29T16:49:04.245 INFO:teuthology.orchestra.run.smithi078.stdout:Generating grub configuration file ... 2023-07-29T16:49:04.603 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.4.0-155-generic 2023-07-29T16:49:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.4.0-155-generic 2023-07-29T16:49:05.040 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-29T16:49:05.051 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-29T16:49:05.404 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub' 2023-07-29T16:49:05.416 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-29T16:49:05.426 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2023-07-29T16:49:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.4.0-155-generic 2023-07-29T16:49:05.688 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.4.0-155-generic 2023-07-29T16:49:06.049 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-29T16:49:06.060 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-29T16:49:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:done 2023-07-29T16:49:07.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2023-07-29T16:49:07.717 INFO:teuthology.orchestra.run.smithi078.stdout:done 2023-07-29T16:49:08.224 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2023-07-29T16:49:08.272 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2023-07-29T16:49:08.272 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2023-07-29T16:49:08.272 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2023-07-29T16:49:08.272 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 19 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.4.0.155.151 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-29T16:49:08.273 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.4.0-155-generic, linux-modules-extra-5.4.0-155-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-29T16:49:08.274 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2023-07-29T16:49:08.274 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2023-07-29T16:49:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2023-07-29T16:49:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2023-07-29T16:49:08.274 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-155-generic 2023-07-29T16:49:08.274 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-29T16:49:08.275 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-29T16:49:08.275 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-07-29T16:49:08.275 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2023-07-29T16:49:08.275 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2023-07-29T16:49:08.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2023-07-29T16:49:08.512 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-29T16:49:08.525 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-07-29T16:49:08.548 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-07-29T16:49:08.624 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-07-29T16:49:10.237 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2023-07-29T16:49:10.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-29T16:49:10.333 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2023-07-29T16:49:10.508 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2023-07-29T16:49:10.508 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2023-07-29T16:49:10.708 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic is already the newest version (5.4.0.155.151). 2023-07-29T16:49:10.709 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 226 not upgraded. 2023-07-29T16:49:10.710 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2023-07-29T16:49:10.726 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2023-07-29T16:49:10.726 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 19 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2023-07-29T16:49:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.4.0.155.151 2023-07-29T16:49:10.728 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-29T16:49:10.728 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.4.0-155-generic, linux-modules-extra-5.4.0-155-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-29T16:49:10.728 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2023-07-29T16:49:10.729 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2023-07-29T16:49:10.729 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2023-07-29T16:49:10.729 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2023-07-29T16:49:10.729 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2023-07-29T16:49:10.734 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.TFPevpcaDA 2023-07-29T16:49:10.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub/grub.cfg /tmp/tmp.TFPevpcaDA 2023-07-29T16:49:10.789 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.TFPevpcaDA 2023-07-29T16:49:10.827 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.TFPevpcaDA is 9KB 2023-07-29T16:49:10.840 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.TFPevpcaDA 2023-07-29T16:49:10.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-07-29T16:49:10.902 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:49:10.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-07-29T16:49:10.903 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-07-29T16:49:10.936 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-155-generic 2023-07-29T16:49:10.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo update-grub 2023-07-29T16:49:12.274 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub' 2023-07-29T16:49:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-29T16:49:12.296 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2023-07-29T16:49:12.580 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.4.0-155-generic 2023-07-29T16:49:12.592 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.4.0-155-generic 2023-07-29T16:49:12.982 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-29T16:49:12.994 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-29T16:49:14.682 INFO:teuthology.orchestra.run.smithi078.stderr:done 2023-07-29T16:49:14.684 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2023-07-29T16:49:37.056 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-07-29T16:49:37.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:49:37.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:49:44.692 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-07-29T16:49:44.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:49:44.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:49:55.568 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-07-29T16:50:03.248 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-07-29T16:50:04.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:50:04.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:50:07.632 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-07-29T16:50:12.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:50:12.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:50:15.312 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-07-29T16:50:19.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:50:19.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:50:27.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:50:27.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:50:38.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-07-29T16:50:45.744 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-07-29T16:50:53.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:50:53.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:50:53.281 DEBUG:teuthology.orchestra.run.smithi129:> true 2023-07-29T16:50:53.839 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:50:53.839 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-155-generic"... 2023-07-29T16:50:53.840 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2023-07-29T16:50:53.846 INFO:teuthology.orchestra.run.smithi129.stdout:5.4.0-155-generic 2023-07-29T16:50:53.846 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-155-generic vs 5.4.0-155-generic 2023-07-29T16:50:53.846 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-29T16:50:53.846 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-07-29T16:50:54.847 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-29T16:50:54.848 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-29T16:50:54.936 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2023-07-29T16:50:54.962 DEBUG:teuthology.parallel:result is None 2023-07-29T16:51:00.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:51:00.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:51:00.954 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-07-29T16:51:01.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:51:01.470 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-155-generic"... 2023-07-29T16:51:01.471 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-07-29T16:51:01.477 INFO:teuthology.orchestra.run.smithi078.stdout:5.4.0-155-generic 2023-07-29T16:51:01.478 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-155-generic vs 5.4.0-155-generic 2023-07-29T16:51:01.478 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-29T16:51:01.478 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-07-29T16:51:02.479 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-29T16:51:02.479 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-29T16:51:02.565 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2023-07-29T16:51:02.593 DEBUG:teuthology.parallel:result is None 2023-07-29T16:51:02.594 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-29T16:51:02.601 INFO:teuthology.task.internal:Creating test directory... 2023-07-29T16:51:02.601 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-29T16:51:02.603 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-29T16:51:02.611 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-29T16:51:02.635 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-29T16:51:02.669 INFO:teuthology.task.internal:Creating archive directory... 2023-07-29T16:51:02.669 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-29T16:51:02.672 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-29T16:51:02.705 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-29T16:51:02.713 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-29T16:51:02.713 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-29T16:51:02.741 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-29T16:51:02.769 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T16:51:02.774 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T16:51:02.780 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T16:51:02.785 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T16:51:02.787 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-29T16:51:02.798 INFO:teuthology.task.internal:Configuring sudo... 2023-07-29T16:51:02.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-29T16:51:02.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-29T16:51:02.820 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-29T16:51:02.857 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-29T16:51:02.857 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-29T16:51:02.860 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-29T16:51:02.869 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T16:51:02.876 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T16:51:02.926 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:51:02.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-29T16:51:02.954 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T16:51:02.961 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T16:51:02.968 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:51:02.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-29T16:51:03.034 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2023-07-29T16:51:03.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2023-07-29T16:51:03.085 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-29T16:51:03.093 INFO:teuthology.task.internal:Starting timer... 2023-07-29T16:51:03.093 INFO:teuthology.run_tasks:Running task pcp... 2023-07-29T16:51:03.140 INFO:teuthology.run_tasks:Running task selinux... 2023-07-29T16:51:03.173 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2023-07-29T16:51:03.174 DEBUG:teuthology.task.selinux:Excluding smithi129: OS 'ubuntu' does not support SELinux 2023-07-29T16:51:03.174 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-29T16:51:03.174 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-07-29T16:51:03.174 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-29T16:51:03.175 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-29T16:51:03.204 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-29T16:51:03.211 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-29T16:51:03.381 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-29T16:51:03.402 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-29T16:51:03.403 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-29T16:56:45.624 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2023-07-29T16:56:45.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:56:45.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:56:45.675 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-07-29T16:56:45.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-29T16:56:45.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:56:45.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T16:56:45.846 DEBUG:teuthology.orchestra.run.smithi129:> true 2023-07-29T16:56:45.965 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-07-29T16:56:45.966 INFO:teuthology.run_tasks:Running task clock... 2023-07-29T16:56:45.977 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-29T16:56:45.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-29T16:56:45.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T16:56:45.981 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-29T16:56:45.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T16:56:46.013 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-07-29T16:56:46.014 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Command line: ntpd -gq 2023-07-29T16:56:46.015 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: proto: precision = 0.116 usec (-23) 2023-07-29T16:56:46.016 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-07-29T16:56:46.016 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Command line: ntpd -gq 2023-07-29T16:56:46.016 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-29T16:56:46.017 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-29T16:56:46.017 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2023-07-29T16:56:46.018 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: restrict ::: KOD does nothing without LIMITED. 2023-07-29T16:56:46.018 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listen and drop on 0 v6wildcard [::]:123 2023-07-29T16:56:46.018 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-07-29T16:56:46.019 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listen normally on 2 lo 127.0.0.1:123 2023-07-29T16:56:46.019 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2023-07-29T16:56:46.019 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listen normally on 4 lo [::1]:123 2023-07-29T16:56:46.019 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2023-07-29T16:56:46.019 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:46 ntpd[22228]: Listening on routing socket on fd #22 for interface updates 2023-07-29T16:56:46.020 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: proto: precision = 0.078 usec (-24) 2023-07-29T16:56:46.020 INFO:teuthology.orchestra.run.smithi129.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-29T16:56:46.021 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-29T16:56:46.021 INFO:teuthology.orchestra.run.smithi129.stderr:restrict ::: KOD does nothing without LIMITED. 2023-07-29T16:56:46.023 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: restrict ::: KOD does nothing without LIMITED. 2023-07-29T16:56:46.024 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listen and drop on 0 v6wildcard [::]:123 2023-07-29T16:56:46.024 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-07-29T16:56:46.025 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listen normally on 2 lo 127.0.0.1:123 2023-07-29T16:56:46.025 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listen normally on 3 enp3s0f1 172.21.15.129:123 2023-07-29T16:56:46.025 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listen normally on 4 lo [::1]:123 2023-07-29T16:56:46.025 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7261%5]:123 2023-07-29T16:56:46.025 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:46 ntpd[22222]: Listening on routing socket on fd #22 for interface updates 2023-07-29T16:56:47.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:47.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:47.020 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.020 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.020 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:47.020 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:47 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:47.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.024 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.024 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:47.025 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:47.025 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:47.025 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:47.025 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:47.026 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:47 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:49.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:49.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:49.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:49.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:49 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:49.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:49.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:49.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:49.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:49.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:49 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:50.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:50 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:50.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:50 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:50.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:50 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:50.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:50 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:50.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:50 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:50.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:50 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:50 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:50 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:50 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:50 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:50 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:50.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:50 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:51.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:51 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:51.022 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:51.024 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:51.024 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:51 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:52.014 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:52 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:52.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:52 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:52.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:52 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:52.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:52 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:52.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:52 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:52 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:52.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:52 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:52.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:52 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:52.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:52 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:52.017 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:52 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:52.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:52 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:52.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:52 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:53.016 INFO:teuthology.orchestra.run.smithi078.stdout:29 Jul 16:56:53 ntpd[22228]: ntpd: time slew +0.000849 s 2023-07-29T16:56:53.016 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time slew +0.000849s 2023-07-29T16:56:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-07-29T16:56:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:29 Jul 16:56:53 ntpd[22228]: can't open /var/log/ntpstats/loopstats.20230729: Permission denied 2023-07-29T16:56:53.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:53.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:53.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:53.018 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:53.019 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-29T16:56:53.019 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: can't open /var/log/ntpstats/rawstats.20230729: Permission denied 2023-07-29T16:56:53.019 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-29T16:56:53.019 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: can't open /var/log/ntpstats/peerstats.20230729: Permission denied 2023-07-29T16:56:53.019 INFO:teuthology.orchestra.run.smithi129.stdout:29 Jul 16:56:53 ntpd[22222]: ntpd: time slew +0.000269 s 2023-07-29T16:56:53.020 INFO:teuthology.orchestra.run.smithi129.stdout:ntpd: time slew +0.000269s 2023-07-29T16:56:53.020 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-07-29T16:56:53.020 INFO:teuthology.orchestra.run.smithi129.stderr:29 Jul 16:56:53 ntpd[22222]: can't open /var/log/ntpstats/loopstats.20230729: Permission denied 2023-07-29T16:56:53.089 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2023-07-29T16:56:53.089 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2023-07-29T16:56:53.089 INFO:teuthology.orchestra.run.smithi129.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.089 INFO:teuthology.orchestra.run.smithi129.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.089 INFO:teuthology.orchestra.run.smithi129.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.089 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2023-07-29T16:56:53.091 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2023-07-29T16:56:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-29T16:56:53.092 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-29T16:56:53.203 INFO:tasks.cephadm:Config: {'allow_ptrace': False, 'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v16.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'} 2023-07-29T16:56:53.204 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v16.2.0 2023-07-29T16:56:53.213 INFO:tasks.cephadm:Cluster fsid is eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:56:53.214 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-29T16:56:53.214 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.c': '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]', 'mon.b': '172.21.15.129'} 2023-07-29T16:56:53.215 INFO:tasks.cephadm:First mon is mon.a on smithi078 2023-07-29T16:56:53.215 INFO:tasks.cephadm:First mgr is y 2023-07-29T16:56:53.215 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-29T16:56:53.216 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2023-07-29T16:56:53.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2023-07-29T16:56:53.252 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.0)... 2023-07-29T16:56:53.253 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-29T16:56:53.456 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-rw-r-- 1 ubuntu ubuntu 288386 Jul 29 16:56 /home/ubuntu/cephtest/cephadm 2023-07-29T16:56:53.457 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-29T16:56:53.639 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-rw-r-- 1 ubuntu ubuntu 288386 Jul 29 16:56 /home/ubuntu/cephtest/cephadm 2023-07-29T16:56:53.639 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-29T16:56:53.649 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-29T16:56:53.674 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v16.2.0 on all hosts... 2023-07-29T16:56:53.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2023-07-29T16:56:53.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2023-07-29T16:56:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2023-07-29T16:56:53.960 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2023-07-29T16:57:24.294 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-07-29T16:57:24.294 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2023-07-29T16:57:24.295 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2023-07-29T16:57:24.295 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2023-07-29T16:57:24.295 INFO:teuthology.orchestra.run.smithi129.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2023-07-29T16:57:24.295 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2023-07-29T16:57:24.295 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-07-29T16:57:25.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T16:57:25.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2023-07-29T16:57:25.670 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2023-07-29T16:57:25.670 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2023-07-29T16:57:25.671 INFO:teuthology.orchestra.run.smithi078.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2023-07-29T16:57:25.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-07-29T16:57:25.671 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T16:57:25.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2023-07-29T16:57:25.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2023-07-29T16:57:25.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2023-07-29T16:57:25.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2023-07-29T16:57:25.781 INFO:tasks.cephadm:Writing seed config... 2023-07-29T16:57:25.783 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2023-07-29T16:57:25.784 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-29T16:57:25.784 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-29T16:57:25.784 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2023-07-29T16:57:25.784 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-29T16:57:25.785 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-29T16:57:25.785 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-29T16:57:25.785 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-29T16:57:25.785 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-29T16:57:25.786 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:57:25.786 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-29T16:57:25.793 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eb1107b2-2e30-11ee-9b36-001a4aab830c mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-07-29T16:57:25.794 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service 2023-07-29T16:57:25.839 DEBUG:teuthology.orchestra.run.smithi078:mgr.y> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service 2023-07-29T16:57:25.841 INFO:tasks.cephadm:Bootstrapping... 2023-07-29T16:57:25.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 -v bootstrap --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:57:25.996 INFO:teuthology.orchestra.run.smithi078.stderr:Could not locate podman: podman not found 2023-07-29T16:57:25.997 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying podman|docker is present... 2023-07-29T16:57:25.997 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying lvm2 is present... 2023-07-29T16:57:25.997 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying time synchronization is in place... 2023-07-29T16:57:25.997 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chrony.service 2023-07-29T16:57:26.002 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-07-29T16:57:26.003 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chrony.service 2023-07-29T16:57:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.010 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chronyd.service 2023-07-29T16:57:26.016 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-07-29T16:57:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chronyd.service 2023-07-29T16:57:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2023-07-29T16:57:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: masked 2023-07-29T16:57:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active systemd-timesyncd.service 2023-07-29T16:57:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled ntpd.service 2023-07-29T16:57:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-07-29T16:57:26.041 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active ntpd.service 2023-07-29T16:57:26.046 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.047 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled ntp.service 2023-07-29T16:57:26.053 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: enabled 2023-07-29T16:57:26.053 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active ntp.service 2023-07-29T16:57:26.060 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: active 2023-07-29T16:57:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:Unit ntp.service is enabled and running 2023-07-29T16:57:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:Repeating the final host check... 2023-07-29T16:57:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:podman|docker (/usr/bin/docker) is present 2023-07-29T16:57:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl is present 2023-07-29T16:57:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:lvcreate is present 2023-07-29T16:57:26.062 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chrony.service 2023-07-29T16:57:26.067 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-07-29T16:57:26.067 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chrony.service 2023-07-29T16:57:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled chronyd.service 2023-07-29T16:57:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-07-29T16:57:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active chronyd.service 2023-07-29T16:57:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2023-07-29T16:57:26.094 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: masked 2023-07-29T16:57:26.094 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active systemd-timesyncd.service 2023-07-29T16:57:26.099 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.100 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled ntpd.service 2023-07-29T16:57:26.107 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-07-29T16:57:26.108 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active ntpd.service 2023-07-29T16:57:26.114 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: inactive 2023-07-29T16:57:26.115 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-enabled ntp.service 2023-07-29T16:57:26.122 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: enabled 2023-07-29T16:57:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl is-active ntp.service 2023-07-29T16:57:26.129 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: active 2023-07-29T16:57:26.130 INFO:teuthology.orchestra.run.smithi078.stderr:Unit ntp.service is enabled and running 2023-07-29T16:57:26.130 INFO:teuthology.orchestra.run.smithi078.stderr:Host looks OK 2023-07-29T16:57:26.130 INFO:teuthology.orchestra.run.smithi078.stderr:Cluster fsid: eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:26.131 INFO:teuthology.orchestra.run.smithi078.stderr:Acquiring lock 140001085345744 on /run/cephadm/eb1107b2-2e30-11ee-9b36-001a4aab830c.lock 2023-07-29T16:57:26.131 INFO:teuthology.orchestra.run.smithi078.stderr:Lock 140001085345744 acquired on /run/cephadm/eb1107b2-2e30-11ee-9b36-001a4aab830c.lock 2023-07-29T16:57:26.131 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying IP 172.21.15.78 port 3300 ... 2023-07-29T16:57:26.132 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying IP 172.21.15.78 port 6789 ... 2023-07-29T16:57:26.132 INFO:teuthology.orchestra.run.smithi078.stderr:Base mon IP is 172.21.15.78, final addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-07-29T16:57:26.132 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/sbin/ip route ls 2023-07-29T16:57:26.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2023-07-29T16:57:26.136 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.78 metric 100 2023-07-29T16:57:26.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-07-29T16:57:26.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 2023-07-29T16:57:26.137 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.78 metric 100 2023-07-29T16:57:26.137 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/sbin/ip -6 route ls 2023-07-29T16:57:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-07-29T16:57:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-07-29T16:57:26.141 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1799sec pref medium 2023-07-29T16:57:26.142 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/sbin/ip -6 addr ls 2023-07-29T16:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-29T16:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2023-07-29T16:57:26.145 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-07-29T16:57:26.146 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-29T16:57:26.146 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2023-07-29T16:57:26.146 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-07-29T16:57:26.146 INFO:teuthology.orchestra.run.smithi078.stderr:Mon IP 172.21.15.78 is in CIDR network 172.21.0.0/20 2023-07-29T16:57:26.147 INFO:teuthology.orchestra.run.smithi078.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-29T16:57:26.147 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2023-07-29T16:57:26.148 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker pull quay.io/ceph/ceph:v16.2.0 2023-07-29T16:57:26.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T16:57:26.342 INFO:journalctl@ceph.mgr.y.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T16:57:26.725 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/docker: v16.2.0: Pulling from ceph/ceph 2023-07-29T16:57:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/docker: Digest: sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a 2023-07-29T16:57:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/docker: Status: Image is up to date for quay.io/ceph/ceph:v16.2.0 2023-07-29T16:57:26.896 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/docker: quay.io/ceph/ceph:v16.2.0 2023-07-29T16:57:26.899 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --version 2023-07-29T16:57:27.928 INFO:teuthology.orchestra.run.smithi078.stderr:ceph: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2023-07-29T16:57:28.489 INFO:teuthology.orchestra.run.smithi078.stderr:Ceph version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2023-07-29T16:57:28.490 INFO:teuthology.orchestra.run.smithi078.stderr:Extracting ceph user uid/gid from container image... 2023-07-29T16:57:28.490 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 -c %u %g /var/lib/ceph 2023-07-29T16:57:29.335 INFO:teuthology.orchestra.run.smithi078.stderr:stat: 167 167 2023-07-29T16:57:29.924 INFO:teuthology.orchestra.run.smithi078.stderr:Creating initial keys... 2023-07-29T16:57:29.924 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2023-07-29T16:57:30.752 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-authtool: AQD6RMVkq8RvLBAA8zJE/ZnsrOcJWC3NTHQ+wg== 2023-07-29T16:57:31.317 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2023-07-29T16:57:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-authtool: AQD8RMVk0RuFBxAAb8kMY2slM0IMaGW6ttUdrw== 2023-07-29T16:57:32.755 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2023-07-29T16:57:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-authtool: AQD9RMVkVvrcJRAAzpHDRs7Gw989Uf3QJ8x9eA== 2023-07-29T16:57:34.237 INFO:teuthology.orchestra.run.smithi078.stderr:Creating initial monmap... 2023-07-29T16:57:34.237 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpjph7iz5q:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --create --clobber --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c --addv a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] /tmp/monmap 2023-07-29T16:57:35.084 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-29T16:57:35.084 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:35.084 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-29T16:57:35.633 INFO:teuthology.orchestra.run.smithi078.stderr:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-29T16:57:35.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: set fsid to eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:35.633 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-29T16:57:35.634 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:57:35.634 INFO:teuthology.orchestra.run.smithi078.stderr:Creating mon... 2023-07-29T16:57:35.636 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpz8t9c92u:/tmp/keyring:z -v /tmp/ceph-tmpjph7iz5q:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --mkfs -i a --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-07-29T16:57:36.509 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-29T16:57:36.510 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 1 imported monmap: 2023-07-29T16:57:36.510 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: epoch 0 2023-07-29T16:57:36.510 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:36.510 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: last_changed 2023-07-29T16:57:35.076444+0000 2023-07-29T16:57:36.513 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: created 2023-07-29T16:57:35.076444+0000 2023-07-29T16:57:36.513 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-07-29T16:57:36.513 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 0 /usr/bin/ceph-mon: set fsid to eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: RocksDB version: 6.8.1 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-07-29T16:57:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Compile date Mar 30 2021 2023-07-29T16:57:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: DB SUMMARY 2023-07-29T16:57:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2023-07-29T16:57:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2023-07-29T16:57:36.523 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.523 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.error_if_exists: 0 2023-07-29T16:57:36.523 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.create_if_missing: 1 2023-07-29T16:57:36.523 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.paranoid_checks: 1 2023-07-29T16:57:36.523 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.env: 0x55af3bf611c0 2023-07-29T16:57:36.523 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.fs: Posix File System 2023-07-29T16:57:36.524 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.info_log: 0x55af3d950f40 2023-07-29T16:57:36.524 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-29T16:57:36.524 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.statistics: (nil) 2023-07-29T16:57:36.524 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.use_fsync: 0 2023-07-29T16:57:36.524 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_log_file_size: 0 2023-07-29T16:57:36.524 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.allow_fallocate: 1 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.use_direct_reads: 0 2023-07-29T16:57:36.525 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.create_missing_column_families: 0 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.db_log_dir: 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_subcompactions: 1 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_background_flushes: -1 2023-07-29T16:57:36.526 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.advise_random_on_open: 1 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.write_buffer_manager: 0x55af3d95b410 2023-07-29T16:57:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.rate_limiter: (nil) 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-29T16:57:36.528 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.unordered_write: 0 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T16:57:36.529 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.row_cache: None 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.wal_filter: None 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.preserve_deletes: 0 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.two_write_queues: 0 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.manual_wal_flush: 0 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.atomic_flush: 0 2023-07-29T16:57:36.530 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.log_readahead_size: 0 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_background_jobs: 2 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_background_compactions: -1 2023-07-29T16:57:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T16:57:36.532 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T16:57:36.532 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T16:57:36.532 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_total_wal_size: 0 2023-07-29T16:57:36.532 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.max_open_files: -1 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.bytes_per_sync: 0 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T16:57:36.533 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Compression algorithms supported: 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kZSTD supported: 0 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kXpressCompression supported: 0 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-29T16:57:36.534 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kLZ4Compression supported: 1 2023-07-29T16:57:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kBZip2Compression supported: 0 2023-07-29T16:57:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kZlibCompression supported: 1 2023-07-29T16:57:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: kSnappyCompression supported: 1 2023-07-29T16:57:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.497+0000 7fceb541f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T16:57:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-07-29T16:57:36.535 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.536 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2023-07-29T16:57:36.536 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.536 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-07-29T16:57:36.536 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.537 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T16:57:36.537 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.merge_operator: 2023-07-29T16:57:36.537 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_filter: None 2023-07-29T16:57:36.537 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_filter_factory: None 2023-07-29T16:57:36.554 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T16:57:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T16:57:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55af3d8680d0) 2023-07-29T16:57:36.555 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-07-29T16:57:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T16:57:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T16:57:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-07-29T16:57:36.556 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: index_type: 0 2023-07-29T16:57:36.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-07-29T16:57:36.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-07-29T16:57:36.557 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-07-29T16:57:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-07-29T16:57:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: checksum: 1 2023-07-29T16:57:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-07-29T16:57:36.558 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache: 0x55af3d8a0d10 2023-07-29T16:57:36.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-07-29T16:57:36.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-07-29T16:57:36.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-07-29T16:57:36.559 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-07-29T16:57:36.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-07-29T16:57:36.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-07-29T16:57:36.560 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-07-29T16:57:36.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-07-29T16:57:36.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-07-29T16:57:36.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-07-29T16:57:36.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-07-29T16:57:36.561 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-07-29T16:57:36.562 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: format_version: 2 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: block_align: 0 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression: NoCompression 2023-07-29T16:57:36.563 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.num_levels: 7 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T16:57:36.564 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression_opts.level: 32767 2023-07-29T16:57:36.565 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compression_opts.enabled: false 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-29T16:57:36.566 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T16:57:36.567 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T16:57:36.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T16:57:36.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T16:57:36.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T16:57:36.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T16:57:36.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T16:57:36.568 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.arena_block_size: 4194304 2023-07-29T16:57:36.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T16:57:36.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T16:57:36.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-07-29T16:57:36.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-29T16:57:36.569 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T16:57:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.table_properties_collectors: 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.inplace_update_support: 0 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T16:57:36.571 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T16:57:36.572 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T16:57:36.572 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.bloom_locality: 0 2023-07-29T16:57:36.572 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.max_successive_merges: 0 2023-07-29T16:57:36.572 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T16:57:36.572 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-29T16:57:36.573 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.force_consistency_checks: 0 2023-07-29T16:57:36.573 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-29T16:57:36.573 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.ttl: 2592000 2023-07-29T16:57:36.573 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.501+0000 7fceb541f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T16:57:36.573 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.505+0000 7fceb541f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T16:57:36.574 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.574 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.505+0000 7fceb541f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-07-29T16:57:36.574 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.574 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.505+0000 7fceb541f700 4 rocksdb: DB pointer 0x55af3d965800 2023-07-29T16:57:36.575 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.505+0000 7fce9e2ba700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-07-29T16:57:36.575 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.505+0000 7fce9e2ba700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-07-29T16:57:36.575 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-07-29T16:57:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:57:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T16:57:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:57:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:57:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T16:57:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-29T16:57:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:36.578 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:57:36.579 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.580 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:57:36.581 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.513+0000 7fceb541f700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-07-29T16:57:36.582 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.513+0000 7fceb541f700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-07-29T16:57:36.583 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph-mon: debug 2023-07-29T16:57:36.513+0000 7fceb541f700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2023-07-29T16:57:37.114 INFO:teuthology.orchestra.run.smithi078.stderr:create mon.a on 2023-07-29T16:57:37.115 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:37.122 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl disable ceph.target 2023-07-29T16:57:37.320 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph.target 2023-07-29T16:57:37.327 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2023-07-29T16:57:37.489 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph.target 2023-07-29T16:57:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target 2023-07-29T16:57:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target -> /etc/systemd/system/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target. 2023-07-29T16:57:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target -> /etc/systemd/system/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target. 2023-07-29T16:57:37.688 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target 2023-07-29T16:57:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl daemon-reload 2023-07-29T16:57:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:57:37.893 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl reset-failed ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:57:37.898 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to reset failed state of unit ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service: Unit ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service not loaded. 2023-07-29T16:57:37.899 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:57:37.906 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target.wants/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service -> /etc/systemd/system/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@.service. 2023-07-29T16:57:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:57:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld does not appear to be present 2023-07-29T16:57:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to enable service . firewalld.service is not available 2023-07-29T16:57:38.104 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-07-29T16:57:38.104 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mon to start... 2023-07-29T16:57:38.104 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mon... 2023-07-29T16:57:38.105 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status 2023-07-29T16:57:39.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:39 smithi078 bash[23250]: cluster 2023-07-29T16:57:39.373006+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-07-29T16:57:39.743 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: cluster: 2023-07-29T16:57:39.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: id: eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:39.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-07-29T16:57:39.744 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:39.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: services: 2023-07-29T16:57:39.745 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.350578s) 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mgr: no daemons active 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: data: 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-07-29T16:57:39.746 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-07-29T16:57:39.747 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: pgs: 2023-07-29T16:57:39.747 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:mon is available 2023-07-29T16:57:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:Assimilating anything we can from ceph.conf... 2023-07-29T16:57:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z -v /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2023-07-29T16:57:40.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:40 smithi078 bash[23250]: cluster 2023-07-29T16:57:39.385388+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-07-29T16:57:40.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:40 smithi078 bash[23250]: cluster 2023-07-29T16:57:39.385655+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} 2023-07-29T16:57:40.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:40 smithi078 bash[23250]: cluster 2023-07-29T16:57:39.394525+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-07-29T16:57:40.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:40 smithi078 bash[23250]: cluster 2023-07-29T16:57:39.404050+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-07-29T16:57:40.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:40 smithi078 bash[23250]: cluster 2023-07-29T16:57:39.405462+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-07-29T16:57:40.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:40 smithi078 bash[23250]: audit 2023-07-29T16:57:39.735897+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.78:0/618707117' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-29T16:57:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [global] 2023-07-29T16:57:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: fsid = eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:41.500 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: log_to_journald = false 2023-07-29T16:57:41.500 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-07-29T16:57:41.500 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-07-29T16:57:41.501 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-07-29T16:57:41.501 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-07-29T16:57:41.501 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-07-29T16:57:41.501 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [mgr] 2023-07-29T16:57:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = True 2023-07-29T16:57:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-07-29T16:57:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [mon] 2023-07-29T16:57:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim = false 2023-07-29T16:57:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-29T16:57:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: [osd] 2023-07-29T16:57:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-07-29T16:57:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-07-29T16:57:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-07-29T16:57:41.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:41 smithi078 bash[23250]: audit 2023-07-29T16:57:41.483633+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.78:0/2721781673' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-29T16:57:41.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:41 smithi078 bash[23250]: audit 2023-07-29T16:57:41.488966+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.78:0/2721781673' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-29T16:57:42.114 INFO:teuthology.orchestra.run.smithi078.stderr:Generating new minimal ceph.conf... 2023-07-29T16:57:42.114 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z -v /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2023-07-29T16:57:43.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:43 smithi078 bash[23250]: audit 2023-07-29T16:57:43.221365+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.78:0/1053034086' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:57:43.845 INFO:teuthology.orchestra.run.smithi078.stderr:Restarting the monitor... 2023-07-29T16:57:43.845 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl restart ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:57:44.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:43 smithi078 systemd[1]: Stopping Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T16:57:44.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:43 smithi078 bash[23250]: debug 2023-07-29T16:57:43.933+0000 7f4a15d20700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T16:57:44.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:43 smithi078 bash[23250]: debug 2023-07-29T16:57:43.933+0000 7f4a15d20700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-29T16:57:44.737 INFO:teuthology.orchestra.run.smithi078.stderr:Setting mon public_network to 172.21.0.0/20 2023-07-29T16:57:44.738 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mon public_network 172.21.0.0/20 2023-07-29T16:57:45.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:44 smithi078 docker[23641]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mon.a 2023-07-29T16:57:45.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:44 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service: Succeeded. 2023-07-29T16:57:45.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:44 smithi078 systemd[1]: Stopped Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T16:57:45.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:44 smithi078 systemd[1]: Started Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T16:57:46.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:45 smithi078 bash[23704]: debug 2023-07-29T16:57:45.985+0000 7f73b214e700 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-29T16:57:46.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:45 smithi078 bash[23704]: debug 2023-07-29T16:57:45.985+0000 7f73b214e700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 6 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:45 smithi078 bash[23704]: debug 2023-07-29T16:57:45.985+0000 7f73b214e700 0 pidfile_write: ignore empty --pid-file 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 0 load: jerasure load: lrc load: isa 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: RocksDB version: 6.8.1 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Compile date Mar 30 2021 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: DB SUMMARY 2023-07-29T16:57:46.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: CURRENT file: CURRENT 2023-07-29T16:57:46.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: IDENTITY file: IDENTITY 2023-07-29T16:57:46.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-07-29T16:57:46.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2023-07-29T16:57:46.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 83076 ; 2023-07-29T16:57:46.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.error_if_exists: 0 2023-07-29T16:57:46.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.create_if_missing: 0 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.paranoid_checks: 1 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.env: 0x564883c541c0 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.fs: Posix File System 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.info_log: 0x564885706fe0 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.statistics: (nil) 2023-07-29T16:57:46.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.use_fsync: 0 2023-07-29T16:57:46.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_log_file_size: 0 2023-07-29T16:57:46.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T16:57:46.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T16:57:46.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-29T16:57:46.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-29T16:57:46.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.allow_fallocate: 1 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.use_direct_reads: 0 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.create_missing_column_families: 0 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.db_log_dir: 2023-07-29T16:57:46.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_subcompactions: 1 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_background_flushes: -1 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T16:57:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.advise_random_on_open: 1 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.write_buffer_manager: 0x56488579de90 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.rate_limiter: (nil) 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.unordered_write: 0 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T16:57:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.row_cache: None 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.wal_filter: None 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.preserve_deletes: 0 2023-07-29T16:57:46.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.two_write_queues: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.manual_wal_flush: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.atomic_flush: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.log_readahead_size: 0 2023-07-29T16:57:46.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_background_jobs: 2 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_background_compactions: -1 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_total_wal_size: 0 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T16:57:46.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_open_files: -1 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bytes_per_sync: 0 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-29T16:57:46.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Compression algorithms supported: 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kZSTD supported: 0 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kXpressCompression supported: 0 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kLZ4Compression supported: 1 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kBZip2Compression supported: 0 2023-07-29T16:57:46.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kZlibCompression supported: 1 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: kSnappyCompression supported: 1 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.merge_operator: 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_filter: None 2023-07-29T16:57:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_filter_factory: None 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5648856aa0d8) 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cache_index_and_filter_blocks: 1 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: pin_top_level_index_and_filter: 1 2023-07-29T16:57:46.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: index_type: 0 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: data_block_index_type: 0 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: index_shortening: 1 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: data_block_hash_table_util_ratio: 0.750000 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: hash_index_allow_collision: 1 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: checksum: 1 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: no_block_cache: 0 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_cache: 0x5648856e2f10 2023-07-29T16:57:46.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_cache_name: BinnedLRUCache 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_cache_options: 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: capacity : 536870912 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: num_shard_bits : 4 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: strict_capacity_limit : 0 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: high_pri_pool_ratio: 0.000 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_cache_compressed: (nil) 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: persistent_cache: (nil) 2023-07-29T16:57:46.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_size: 4096 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_size_deviation: 10 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_restart_interval: 16 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: index_block_restart_interval: 1 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: metadata_block_size: 4096 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: partition_filters: 0 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: use_delta_encoding: 1 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: filter_policy: rocksdb.BuiltinBloomFilter 2023-07-29T16:57:46.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: whole_key_filtering: 1 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: verify_compression: 0 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: read_amp_bytes_per_bit: 0 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: format_version: 2 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: enable_index_compression: 1 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: block_align: 0 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-29T16:57:46.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression: NoCompression 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.num_levels: 7 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T16:57:46.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T16:57:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression_opts.level: 32767 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compression_opts.enabled: false 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T16:57:46.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T16:57:46.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.arena_block_size: 4194304 2023-07-29T16:57:46.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T16:57:46.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.table_properties_collectors: 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.inplace_update_support: 0 2023-07-29T16:57:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.bloom_locality: 0 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.max_successive_merges: 0 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-29T16:57:46.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.force_consistency_checks: 0 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.ttl: 2592000 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:45.997+0000 7f73b214e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.005+0000 7f73b214e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.005+0000 7f73b214e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.005+0000 7f73b214e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690649866011365, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-07-29T16:57:46.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.005+0000 7f73b214e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.009+0000 7f73b214e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.009+0000 7f73b214e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690649866013290, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 80044, "table_properties": {"data_size": 78302, "index_size": 347, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 8979, "raw_average_key_size": 46, "raw_value_size": 73061, "raw_average_value_size": 376, "num_data_blocks": 11, "num_entries": 194, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1690649866, "oldest_key_time": 3, "file_creation_time": 0}} 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.009+0000 7f73b214e700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.009+0000 7f73b214e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690649866014317, "job": 1, "event": "recovery_finished"} 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.013+0000 7f73b214e700 4 rocksdb: DB pointer 0x5648857a7800 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.013+0000 7f7398f68700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-07-29T16:57:46.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.013+0000 7f7398f68700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** DB Stats ** 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-07-29T16:57:46.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** Compaction Stats [default] ** 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: L0 2/0 79.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Sum 2/0 79.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** Compaction Stats [default] ** 2023-07-29T16:57:46.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:57:46.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Cumulative compaction: 0.00 GB write, 5.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Interval compaction: 0.00 GB write, 5.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** Compaction Stats [default] ** 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:46.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: L0 2/0 79.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Sum 2/0 79.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** Compaction Stats [default] ** 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 64.9 0.00 0.00 1 0.001 0 0 2023-07-29T16:57:46.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Cumulative compaction: 0.00 GB write, 5.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:57:46.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.013+0000 7f73b214e700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 1 mon.a@-1(???) e1 preinit fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 0 mon.a@-1(???).mds e1 new map 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 0 mon.a@-1(???).mds e1 print_map 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: e1 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T16:57:46.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: legacy client fscid: -1 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: No filesystems configured 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requiresaudit 2023-07-29T16:57:46.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: 2023-07-debug 2023-07-29T16:57:46.017+0000 7f73b214e700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: 29T16:57:43.221365+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.78:0/1053034086' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expand map: {default=false} 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta from 'false' to 'false' 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expanded map: {default=false} 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expand map: {default=info} 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta from 'info' to 'info' 2023-07-29T16:57:46.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expanded map: {default=info} 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expand map: {default=daemon} 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expanded map: {default=daemon} 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expand map: {default=debug} 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta from 'debug' to 'debug' 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: debug 2023-07-29T16:57:46.017+0000 7f73b214e700 20 expand_channel_meta expanded map: {default=debug} 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cluster 2023-07-29T16:57:46.024506+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-07-29T16:57:46.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cluster 2023-07-29T16:57:46.024607+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} 2023-07-29T16:57:46.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cluster 2023-07-29T16:57:46.024660+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2023-07-29T16:57:46.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cluster 2023-07-29T16:57:46.024696+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-07-29T16:57:46.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:46 smithi078 bash[23704]: cluster 2023-07-29T16:57:46.025151+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-07-29T16:57:46.939 INFO:teuthology.orchestra.run.smithi078.stderr:Wrote config to /etc/ceph/ceph.conf 2023-07-29T16:57:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:57:46.940 INFO:teuthology.orchestra.run.smithi078.stderr:Creating mgr... 2023-07-29T16:57:46.941 INFO:teuthology.orchestra.run.smithi078.stderr:Verifying port 9283 ... 2023-07-29T16:57:46.943 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl daemon-reload 2023-07-29T16:57:47.141 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y 2023-07-29T16:57:47.154 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl reset-failed ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y 2023-07-29T16:57:47.160 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Failed to reset failed state of unit ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service: Unit ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service not loaded. 2023-07-29T16:57:47.161 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl enable ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y 2023-07-29T16:57:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:systemctl: Created symlink /etc/systemd/system/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c.target.wants/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service -> /etc/systemd/system/ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@.service. 2023-07-29T16:57:47.360 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: systemctl start ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y 2023-07-29T16:57:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld does not appear to be present 2023-07-29T16:57:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to enable service . firewalld.service is not available 2023-07-29T16:57:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-07-29T16:57:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld does not appear to be present 2023-07-29T16:57:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-07-29T16:57:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr to start... 2023-07-29T16:57:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr... 2023-07-29T16:57:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2023-07-29T16:57:47.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:47 smithi078 bash[23704]: audit 2023-07-29T16:57:46.274609+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.78:0/2235696958' entity='client.admin' 2023-07-29T16:57:49.057 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:49.057 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:57:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsid": "eb1107b2-2e30-11ee-9b36-001a4aab830c", 2023-07-29T16:57:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "health": { 2023-07-29T16:57:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-07-29T16:57:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "checks": {}, 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mutes": [] 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum": [ 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 0 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:49.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_names": [ 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "a" 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_age": 3, 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "monmap": { 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-07-29T16:57:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_mons": 1 2023-07-29T16:57:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osdmap": { 2023-07-29T16:57:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_osds": 0, 2023-07-29T16:57:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-07-29T16:57:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-07-29T16:57:49.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-07-29T16:57:49.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-07-29T16:57:49.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-07-29T16:57:49.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgmap": { 2023-07-29T16:57:49.068 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pools": 0, 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_objects": 0, 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-07-29T16:57:49.069 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_total": 0 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsmap": { 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "by_rank": [], 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "up:standby": 0 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.070 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap": { 2023-07-29T16:57:49.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": false, 2023-07-29T16:57:49.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-07-29T16:57:49.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modules": [ 2023-07-29T16:57:49.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "iostat", 2023-07-29T16:57:49.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "restful" 2023-07-29T16:57:49.071 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:49.072 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-07-29T16:57:49.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "servicemap": { 2023-07-29T16:57:49.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:49.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modified": "2023-07-29T16:57:39.389219+0000", 2023-07-29T16:57:49.073 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-07-29T16:57:49.074 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:49.074 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "progress_events": {} 2023-07-29T16:57:49.074 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:57:49.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:49 smithi078 bash[23704]: audit 2023-07-29T16:57:49.049175+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.78:0/2900976999' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:57:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:mgr not available, waiting (1/15)... 2023-07-29T16:57:51.698 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2023-07-29T16:57:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:57:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsid": "eb1107b2-2e30-11ee-9b36-001a4aab830c", 2023-07-29T16:57:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "health": { 2023-07-29T16:57:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-07-29T16:57:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "checks": {}, 2023-07-29T16:57:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mutes": [] 2023-07-29T16:57:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum": [ 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 0 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_names": [ 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "a" 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_age": 6, 2023-07-29T16:57:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "monmap": { 2023-07-29T16:57:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-07-29T16:57:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_mons": 1 2023-07-29T16:57:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osdmap": { 2023-07-29T16:57:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_osds": 0, 2023-07-29T16:57:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-07-29T16:57:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-07-29T16:57:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-07-29T16:57:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-07-29T16:57:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-07-29T16:57:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgmap": { 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pools": 0, 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_objects": 0, 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-07-29T16:57:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_total": 0 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsmap": { 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "by_rank": [], 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "up:standby": 0 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap": { 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": false, 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modules": [ 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "iostat", 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "restful" 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-07-29T16:57:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "servicemap": { 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modified": "2023-07-29T16:57:39.389219+0000", 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "progress_events": {} 2023-07-29T16:57:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:57:53.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:52 smithi078 bash[23704]: audit 2023-07-29T16:57:52.843686+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.78:0/3981533014' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:57:53.419 INFO:teuthology.orchestra.run.smithi078.stderr:mgr not available, waiting (2/15)... 2023-07-29T16:57:54.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: cluster 2023-07-29T16:57:54.569705+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2023-07-29T16:57:54.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: cluster 2023-07-29T16:57:54.576235+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00667961s) 2023-07-29T16:57:54.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.578550+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.579205+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.579814+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.579985+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.580127+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: cluster 2023-07-29T16:57:54.584954+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.601105+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' 2023-07-29T16:57:54.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.608046+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:57:54.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:54 smithi078 bash[23704]: audit 2023-07-29T16:57:54.610589+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T16:57:55.422 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2023-07-29T16:57:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:55 smithi078 bash[23704]: audit 2023-07-29T16:57:54.623884+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' 2023-07-29T16:57:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:55 smithi078 bash[23704]: audit 2023-07-29T16:57:54.631483+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.78:0/2609221731' entity='mgr.y' 2023-07-29T16:57:55.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:55 smithi078 bash[23704]: cluster 2023-07-29T16:57:55.578912+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00938s) 2023-07-29T16:57:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 2023-07-29T16:57:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:57:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsid": "eb1107b2-2e30-11ee-9b36-001a4aab830c", 2023-07-29T16:57:57.014 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "health": { 2023-07-29T16:57:57.015 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-07-29T16:57:57.015 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "checks": {}, 2023-07-29T16:57:57.015 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mutes": [] 2023-07-29T16:57:57.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-07-29T16:57:57.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum": [ 2023-07-29T16:57:57.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 0 2023-07-29T16:57:57.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:57.016 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_names": [ 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "a" 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "monmap": { 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_mons": 1 2023-07-29T16:57:57.017 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osdmap": { 2023-07-29T16:57:57.018 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:57.020 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_osds": 0, 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.021 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgmap": { 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_pools": 0, 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_objects": 0, 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-07-29T16:57:57.022 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "bytes_total": 0 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "fsmap": { 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "by_rank": [], 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "up:standby": 0 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.023 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap": { 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": true, 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modules": [ 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "iostat", 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "restful" 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ], 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-07-29T16:57:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "servicemap": { 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "modified": "2023-07-29T16:57:39.389219+0000", 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "services": {} 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: }, 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "progress_events": {} 2023-07-29T16:57:57.025 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:57:57.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:57 smithi078 bash[23704]: audit 2023-07-29T16:57:57.005033+0000 mon.a (mon.0) 23 : audit [DBG] from='client.? 172.21.15.78:0/4067913682' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:57:57.594 INFO:teuthology.orchestra.run.smithi078.stderr:mgr is available 2023-07-29T16:57:57.595 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config-key set mgr/cephadm/ssh_user root 2023-07-29T16:57:58.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:58 smithi078 bash[23704]: cluster 2023-07-29T16:57:57.036318+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 2s) 2023-07-29T16:57:59.000 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2023-07-29T16:57:59.558 INFO:teuthology.orchestra.run.smithi078.stderr:Enabling cephadm module... 2023-07-29T16:57:59.559 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable cephadm 2023-07-29T16:58:00.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:57:59 smithi078 bash[23704]: audit 2023-07-29T16:57:58.988844+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.78:0/343859126' entity='client.admin' 2023-07-29T16:58:01.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:00 smithi078 bash[23704]: audit 2023-07-29T16:58:00.982524+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.78:0/344377187' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-29T16:58:01.428 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:01 smithi078 bash[23973]: ignoring --setgroup ceph since I am not root 2023-07-29T16:58:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2023-07-29T16:58:02.369 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:01 smithi078 bash[23704]: audit 2023-07-29T16:58:00.998325+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.78:0/344377187' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-29T16:58:02.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:01 smithi078 bash[23704]: cluster 2023-07-29T16:58:00.998425+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 6s) 2023-07-29T16:58:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:58:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 5, 2023-07-29T16:58:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": true, 2023-07-29T16:58:02.987 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "active_name": "y", 2023-07-29T16:58:02.987 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standby": 0 2023-07-29T16:58:02.987 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:58:03.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:02 smithi078 bash[23704]: audit 2023-07-29T16:58:02.978195+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.78:0/1157345714' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-29T16:58:03.570 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for the mgr to restart... 2023-07-29T16:58:03.571 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr epoch 5... 2023-07-29T16:58:03.572 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2023-07-29T16:58:07.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:07 smithi078 bash[23704]: cluster 2023-07-29T16:58:06.837002+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2023-07-29T16:58:07.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:07 smithi078 bash[23704]: cluster 2023-07-29T16:58:06.837769+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2023-07-29T16:58:07.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:07 smithi078 bash[23704]: cluster 2023-07-29T16:58:06.841952+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-07-29T16:58:08.065 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:58:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-07-29T16:58:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "initialized": true 2023-07-29T16:58:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:58:08.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: cluster 2023-07-29T16:58:07.036913+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.199251s) 2023-07-29T16:58:08.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.039758+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.041869+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.043127+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.043379+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.043600+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: cluster 2023-07-29T16:58:07.048121+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.068605+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.075532+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:08.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.081965+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:08.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.220125+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:08.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.221427+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:08.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.223779+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:08.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.228662+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:08.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.249818+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:58:08.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:08 smithi078 bash[23704]: audit 2023-07-29T16:58:07.252208+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T16:58:08.733 INFO:teuthology.orchestra.run.smithi078.stderr:mgr epoch 5 is available 2023-07-29T16:58:08.733 INFO:teuthology.orchestra.run.smithi078.stderr:Setting orchestrator backend to cephadm... 2023-07-29T16:58:08.734 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch set backend cephadm 2023-07-29T16:58:09.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:09 smithi078 bash[23704]: audit 2023-07-29T16:58:08.040556+0000 mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-29T16:58:09.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:09 smithi078 bash[23704]: cluster 2023-07-29T16:58:08.041835+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.20416s) 2023-07-29T16:58:09.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:09 smithi078 bash[23704]: audit 2023-07-29T16:58:08.057917+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-29T16:58:10.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:10 smithi078 bash[23704]: cluster 2023-07-29T16:58:09.053393+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2023-07-29T16:58:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:Generating ssh key... 2023-07-29T16:58:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm generate-key 2023-07-29T16:58:11.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:11 smithi078 bash[23704]: audit 2023-07-29T16:58:10.121359+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:11.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:11 smithi078 bash[23704]: audit 2023-07-29T16:58:10.131114+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:11.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:11 smithi078 bash[23704]: audit 2023-07-29T16:58:10.137465+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:12.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23704]: cephadm 2023-07-29T16:58:12.008885+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2023-07-29T16:58:12.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: Generating public/private rsa key pair. 2023-07-29T16:58:12.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: Your identification has been saved in /tmp/tmpljr8kgk0/key. 2023-07-29T16:58:12.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: Your public key has been saved in /tmp/tmpljr8kgk0/key.pub. 2023-07-29T16:58:12.430 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: The key fingerprint is: 2023-07-29T16:58:12.430 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: SHA256:PbNE9Y+Vq4w+FB2jSpSMgmW8J6i6UOKtnvjlQ9XNvwM ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:58:12.430 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: The key's randomart image is: 2023-07-29T16:58:12.430 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: +---[RSA 3072]----+ 2023-07-29T16:58:12.431 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: | +o o . . | 2023-07-29T16:58:12.431 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: | ..o . + . + .| 2023-07-29T16:58:12.431 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: | . o..o. o +..| 2023-07-29T16:58:12.431 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: | . o...+oo . +.| 2023-07-29T16:58:12.432 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: |. o .o S *.. ...| 2023-07-29T16:58:12.432 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: |.+. . oE=+ . | 2023-07-29T16:58:12.432 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: |o. o. oo + | 2023-07-29T16:58:12.432 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: |+ oo. ..o | 2023-07-29T16:58:12.433 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: |+*. .. ... | 2023-07-29T16:58:12.433 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:12 smithi078 bash[23973]: +----[SHA256]-----+ 2023-07-29T16:58:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm get-pub-key 2023-07-29T16:58:13.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:13 smithi078 bash[23704]: audit 2023-07-29T16:58:12.149749+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:13.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:13 smithi078 bash[23704]: audit 2023-07-29T16:58:12.154236+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+HrRcZ4mgkyXoTZGQkcKpVqjk8H1l822cwMbN1dv7FtXFF0gUTwTRIxT9YFgPWuo9YCaPH4KDCnhalVFyeB/Vb0Tif3PNQYGxrwb0hSy5ttdgdnXCaArIyP0o7zjebxHU88Tn5PLPNLyirwnyeIgtu/zz4drt/fgCbpoXiuHVFKsvKSftHcDLCJXjgopci8UB9k/q1WKJ7bACJLEQ/1oR448j01fxgrm8Cl/Lgt9zZv2wTSlFJ9wyqRWKbmKSdbDRtImzUfp7pF7bKhFrWYhXBQL8qi9S8xM5UYugaNSrJGJWrJ1XOF7sR/2xsihAdC7Z/sTKzkF1Lr3oynR3ZvXSdodeS7IK+90dMOqYuA/dalkvU2Ix3S2FCzrTecRZ9zcbBApmREKTv0GRqZvUOv4ywpJJTsJF3PJDzO6QGGQnZ5hhDEyX6Bczkm3MCWvwuXHOEzC4KgA++NEGQvL5GHdCfMLn7782dTnB6hBXhPdkPSRJEp/WZz+1Cm228wNwg90= ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:58:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-29T16:58:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:Adding key to root@localhost authorized_keys... 2023-07-29T16:58:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:Adding host smithi078... 2023-07-29T16:58:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch host add smithi078 2023-07-29T16:58:15.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:15 smithi078 bash[23704]: audit 2023-07-29T16:58:14.046319+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:16.428 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:16 smithi078 bash[23973]: Warning: Permanently added 'smithi078,172.21.15.78' (ECDSA) to the list of known hosts. 2023-07-29T16:58:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Added host 'smithi078' 2023-07-29T16:58:17.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:17 smithi078 bash[23704]: audit 2023-07-29T16:58:15.960932+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:17.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:17 smithi078 bash[23704]: audit 2023-07-29T16:58:16.699962+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:17.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:17 smithi078 bash[23704]: audit 2023-07-29T16:58:16.701208+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:17.539 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying unmanaged mon service... 2023-07-29T16:58:17.540 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mon --unmanaged 2023-07-29T16:58:18.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:18 smithi078 bash[23704]: cephadm 2023-07-29T16:58:16.700790+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi078 2023-07-29T16:58:19.770 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled mon update... 2023-07-29T16:58:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:Deploying unmanaged mgr service... 2023-07-29T16:58:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mgr --unmanaged 2023-07-29T16:58:20.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:20 smithi078 bash[23704]: audit 2023-07-29T16:58:19.755244+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:20.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:20 smithi078 bash[23704]: cephadm 2023-07-29T16:58:19.756972+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2023-07-29T16:58:20.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:20 smithi078 bash[23704]: audit 2023-07-29T16:58:19.761593+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:20.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:20 smithi078 bash[23704]: audit 2023-07-29T16:58:20.099215+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:20.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:20 smithi078 bash[23704]: audit 2023-07-29T16:58:20.349417+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:22.013 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-07-29T16:58:22.615 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/cephadm/container_init True --force 2023-07-29T16:58:23.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:23 smithi078 bash[23704]: audit 2023-07-29T16:58:21.999081+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:23.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:23 smithi078 bash[23704]: cephadm 2023-07-29T16:58:22.000325+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2023-07-29T16:58:23.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:23 smithi078 bash[23704]: audit 2023-07-29T16:58:22.005295+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:24.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:24 smithi078 bash[23704]: audit 2023-07-29T16:58:23.404535+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.78:0/241276767' entity='mgr.y' 2023-07-29T16:58:24.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:24 smithi078 bash[23704]: audit 2023-07-29T16:58:23.882851+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.78:0/3468998587' entity='client.admin' 2023-07-29T16:58:25.069 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/dashboard/ssl_server_port 8443 2023-07-29T16:58:27.620 INFO:teuthology.orchestra.run.smithi078.stderr:Enabling the dashboard module... 2023-07-29T16:58:27.621 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable dashboard 2023-07-29T16:58:28.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:27 smithi078 bash[23704]: audit 2023-07-29T16:58:26.956091+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.78:0/1110464299' entity='client.admin' 2023-07-29T16:58:28.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:27 smithi078 bash[23704]: cluster 2023-07-29T16:58:27.044584+0000 mgr.y (mgr.14116) 13 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:58:29.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:29 smithi078 bash[23704]: audit 2023-07-29T16:58:29.032546+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.78:0/3856637224' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-29T16:58:29.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:29 smithi078 bash[23704]: cluster 2023-07-29T16:58:29.045027+0000 mgr.y (mgr.14116) 14 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:58:29.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:29 smithi078 bash[23973]: ignoring --setuser ceph since I am not root 2023-07-29T16:58:29.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:29 smithi078 bash[23973]: ignoring --setgroup ceph since I am not root 2023-07-29T16:58:29.785 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2023-07-29T16:58:30.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:30 smithi078 bash[23704]: audit 2023-07-29T16:58:29.089290+0000 mon.a (mon.0) 65 : audit [INF] from='client.? 172.21.15.78:0/3856637224' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-29T16:58:30.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:30 smithi078 bash[23704]: cluster 2023-07-29T16:58:29.089419+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 22s) 2023-07-29T16:58:31.168 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:58:31.169 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "epoch": 9, 2023-07-29T16:58:31.169 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "available": true, 2023-07-29T16:58:31.169 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "active_name": "y", 2023-07-29T16:58:31.169 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "num_standby": 0 2023-07-29T16:58:31.169 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:58:31.775 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for the mgr to restart... 2023-07-29T16:58:31.775 INFO:teuthology.orchestra.run.smithi078.stderr:Waiting for mgr epoch 9... 2023-07-29T16:58:31.775 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2023-07-29T16:58:32.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:32 smithi078 bash[23704]: audit 2023-07-29T16:58:31.160347+0000 mon.a (mon.0) 67 : audit [DBG] from='client.? 172.21.15.78:0/1848493347' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-29T16:58:35.678 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23973]: Warning: Permanently added 'smithi078,172.21.15.78' (ECDSA) to the list of known hosts. 2023-07-29T16:58:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: { 2023-07-29T16:58:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-07-29T16:58:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: "initialized": true 2023-07-29T16:58:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: } 2023-07-29T16:58:36.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: cluster 2023-07-29T16:58:34.973295+0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2023-07-29T16:58:36.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: cluster 2023-07-29T16:58:34.974103+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2023-07-29T16:58:36.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: cluster 2023-07-29T16:58:34.977372+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-07-29T16:58:36.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: cluster 2023-07-29T16:58:35.028123+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0541913s) 2023-07-29T16:58:36.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.031536+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:58:36.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.033327+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T16:58:36.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.034776+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T16:58:36.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.035170+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T16:58:36.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.035487+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T16:58:36.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: cluster 2023-07-29T16:58:35.039741+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2023-07-29T16:58:36.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.192285+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:36.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.197026+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:36.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.235967+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:58:36.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:35 smithi078 bash[23704]: audit 2023-07-29T16:58:35.239735+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T16:58:36.783 INFO:teuthology.orchestra.run.smithi078.stderr:mgr epoch 9 is available 2023-07-29T16:58:36.783 INFO:teuthology.orchestra.run.smithi078.stderr:Generating a dashboard self-signed certificate... 2023-07-29T16:58:36.784 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 dashboard create-self-signed-cert 2023-07-29T16:58:37.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:37 smithi078 bash[23704]: audit 2023-07-29T16:58:36.032571+0000 mgr.y (mgr.14144) 1 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-29T16:58:37.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:37 smithi078 bash[23704]: cluster 2023-07-29T16:58:36.033864+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.05993s) 2023-07-29T16:58:37.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:37 smithi078 bash[23704]: audit 2023-07-29T16:58:36.046444+0000 mgr.y (mgr.14144) 2 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-29T16:58:38.217 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: Self-signed certificate created 2023-07-29T16:58:38.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:38 smithi078 bash[23704]: cluster 2023-07-29T16:58:37.041388+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e12: y(active, since 2s) 2023-07-29T16:58:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:Creating initial admin user... 2023-07-29T16:58:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0s2ttdnl:/tmp/dashboard.pw:z quay.io/ceph/ceph:v16.2.0 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-07-29T16:58:39.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:39 smithi078 bash[23704]: audit 2023-07-29T16:58:38.145314+0000 mgr.y (mgr.14144) 3 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:39.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:39 smithi078 bash[23704]: audit 2023-07-29T16:58:38.206589+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:39.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:39 smithi078 bash[23704]: audit 2023-07-29T16:58:38.209094+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$1/ocu4kDEg5PMOoSwdDYSOTivXzuBYWdx4MooBBmUEDbz/8PGVk.K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1690649920, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-29T16:58:40.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:40 smithi078 bash[23704]: audit 2023-07-29T16:58:39.554295+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:40.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:40 smithi078 bash[23704]: audit 2023-07-29T16:58:40.418104+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:41.267 INFO:teuthology.orchestra.run.smithi078.stderr:Fetching dashboard port number... 2023-07-29T16:58:41.268 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi078 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp458g5cld:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpffzoa4su:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config get mgr mgr/dashboard/ssl_server_port 2023-07-29T16:58:42.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:42 smithi078 bash[23704]: audit 2023-07-29T16:58:40.177309+0000 mgr.y (mgr.14144) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:42.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:42 smithi078 bash[23704]: cluster 2023-07-29T16:58:41.560874+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e13: y(active, since 6s) 2023-07-29T16:58:43.434 INFO:teuthology.orchestra.run.smithi078.stderr:/usr/bin/ceph: 8443 2023-07-29T16:58:43.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:43 smithi078 bash[23704]: audit 2023-07-29T16:58:43.426817+0000 mon.a (mon.0) 89 : audit [DBG] from='client.? 172.21.15.78:0/2656790184' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-29T16:58:44.078 INFO:teuthology.orchestra.run.smithi078.stderr:firewalld does not appear to be present 2023-07-29T16:58:44.079 INFO:teuthology.orchestra.run.smithi078.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-29T16:58:44.082 INFO:teuthology.orchestra.run.smithi078.stderr:Ceph Dashboard is now available at: 2023-07-29T16:58:44.083 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.083 INFO:teuthology.orchestra.run.smithi078.stderr: URL: https://smithi078.front.sepia.ceph.com:8443/ 2023-07-29T16:58:44.083 INFO:teuthology.orchestra.run.smithi078.stderr: User: admin 2023-07-29T16:58:44.084 INFO:teuthology.orchestra.run.smithi078.stderr: Password: k2bc7y52pe 2023-07-29T16:58:44.084 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.086 INFO:teuthology.orchestra.run.smithi078.stderr:You can access the Ceph CLI with: 2023-07-29T16:58:44.086 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.086 INFO:teuthology.orchestra.run.smithi078.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:58:44.087 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.087 INFO:teuthology.orchestra.run.smithi078.stderr:Please consider enabling telemetry to help improve Ceph: 2023-07-29T16:58:44.087 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.087 INFO:teuthology.orchestra.run.smithi078.stderr: ceph telemetry on 2023-07-29T16:58:44.088 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.088 INFO:teuthology.orchestra.run.smithi078.stderr:For more information see: 2023-07-29T16:58:44.088 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.088 INFO:teuthology.orchestra.run.smithi078.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-07-29T16:58:44.089 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T16:58:44.089 INFO:teuthology.orchestra.run.smithi078.stderr:Bootstrap complete. 2023-07-29T16:58:44.089 INFO:teuthology.orchestra.run.smithi078.stderr:Releasing lock 140001085345744 on /run/cephadm/eb1107b2-2e30-11ee-9b36-001a4aab830c.lock 2023-07-29T16:58:44.089 INFO:teuthology.orchestra.run.smithi078.stderr:Lock 140001085345744 released on /run/cephadm/eb1107b2-2e30-11ee-9b36-001a4aab830c.lock 2023-07-29T16:58:44.117 INFO:tasks.cephadm:Fetching config... 2023-07-29T16:58:44.117 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:58:44.117 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-29T16:58:44.121 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-29T16:58:44.122 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:58:44.122 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-29T16:58:44.173 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-29T16:58:44.173 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:58:44.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-29T16:58:44.188 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-29T16:58:44.189 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:58:44.189 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-29T16:58:44.197 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-29T16:58:44.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+HrRcZ4mgkyXoTZGQkcKpVqjk8H1l822cwMbN1dv7FtXFF0gUTwTRIxT9YFgPWuo9YCaPH4KDCnhalVFyeB/Vb0Tif3PNQYGxrwb0hSy5ttdgdnXCaArIyP0o7zjebxHU88Tn5PLPNLyirwnyeIgtu/zz4drt/fgCbpoXiuHVFKsvKSftHcDLCJXjgopci8UB9k/q1WKJ7bACJLEQ/1oR448j01fxgrm8Cl/Lgt9zZv2wTSlFJ9wyqRWKbmKSdbDRtImzUfp7pF7bKhFrWYhXBQL8qi9S8xM5UYugaNSrJGJWrJ1XOF7sR/2xsihAdC7Z/sTKzkF1Lr3oynR3ZvXSdodeS7IK+90dMOqYuA/dalkvU2Ix3S2FCzrTecRZ9zcbBApmREKTv0GRqZvUOv4ywpJJTsJF3PJDzO6QGGQnZ5hhDEyX6Bczkm3MCWvwuXHOEzC4KgA++NEGQvL5GHdCfMLn7782dTnB6hBXhPdkPSRJEp/WZz+1Cm228wNwg90= ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-29T16:58:44.269 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+HrRcZ4mgkyXoTZGQkcKpVqjk8H1l822cwMbN1dv7FtXFF0gUTwTRIxT9YFgPWuo9YCaPH4KDCnhalVFyeB/Vb0Tif3PNQYGxrwb0hSy5ttdgdnXCaArIyP0o7zjebxHU88Tn5PLPNLyirwnyeIgtu/zz4drt/fgCbpoXiuHVFKsvKSftHcDLCJXjgopci8UB9k/q1WKJ7bACJLEQ/1oR448j01fxgrm8Cl/Lgt9zZv2wTSlFJ9wyqRWKbmKSdbDRtImzUfp7pF7bKhFrWYhXBQL8qi9S8xM5UYugaNSrJGJWrJ1XOF7sR/2xsihAdC7Z/sTKzkF1Lr3oynR3ZvXSdodeS7IK+90dMOqYuA/dalkvU2Ix3S2FCzrTecRZ9zcbBApmREKTv0GRqZvUOv4ywpJJTsJF3PJDzO6QGGQnZ5hhDEyX6Bczkm3MCWvwuXHOEzC4KgA++NEGQvL5GHdCfMLn7782dTnB6hBXhPdkPSRJEp/WZz+1Cm228wNwg90= ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:58:44.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+HrRcZ4mgkyXoTZGQkcKpVqjk8H1l822cwMbN1dv7FtXFF0gUTwTRIxT9YFgPWuo9YCaPH4KDCnhalVFyeB/Vb0Tif3PNQYGxrwb0hSy5ttdgdnXCaArIyP0o7zjebxHU88Tn5PLPNLyirwnyeIgtu/zz4drt/fgCbpoXiuHVFKsvKSftHcDLCJXjgopci8UB9k/q1WKJ7bACJLEQ/1oR448j01fxgrm8Cl/Lgt9zZv2wTSlFJ9wyqRWKbmKSdbDRtImzUfp7pF7bKhFrWYhXBQL8qi9S8xM5UYugaNSrJGJWrJ1XOF7sR/2xsihAdC7Z/sTKzkF1Lr3oynR3ZvXSdodeS7IK+90dMOqYuA/dalkvU2Ix3S2FCzrTecRZ9zcbBApmREKTv0GRqZvUOv4ywpJJTsJF3PJDzO6QGGQnZ5hhDEyX6Bczkm3MCWvwuXHOEzC4KgA++NEGQvL5GHdCfMLn7782dTnB6hBXhPdkPSRJEp/WZz+1Cm228wNwg90= ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-29T16:58:44.302 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+HrRcZ4mgkyXoTZGQkcKpVqjk8H1l822cwMbN1dv7FtXFF0gUTwTRIxT9YFgPWuo9YCaPH4KDCnhalVFyeB/Vb0Tif3PNQYGxrwb0hSy5ttdgdnXCaArIyP0o7zjebxHU88Tn5PLPNLyirwnyeIgtu/zz4drt/fgCbpoXiuHVFKsvKSftHcDLCJXjgopci8UB9k/q1WKJ7bACJLEQ/1oR448j01fxgrm8Cl/Lgt9zZv2wTSlFJ9wyqRWKbmKSdbDRtImzUfp7pF7bKhFrWYhXBQL8qi9S8xM5UYugaNSrJGJWrJ1XOF7sR/2xsihAdC7Z/sTKzkF1Lr3oynR3ZvXSdodeS7IK+90dMOqYuA/dalkvU2Ix3S2FCzrTecRZ9zcbBApmREKTv0GRqZvUOv4ywpJJTsJF3PJDzO6QGGQnZ5hhDEyX6Bczkm3MCWvwuXHOEzC4KgA++NEGQvL5GHdCfMLn7782dTnB6hBXhPdkPSRJEp/WZz+1Cm228wNwg90= ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:58:44.313 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2023-07-29T16:58:44.313 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:58:44.314 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2023-07-29T16:58:44.320 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:58:44.321 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:58:44.367 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2023-07-29T16:58:44.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch host add smithi129 2023-07-29T16:58:48.678 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 16:58:48 smithi078 bash[23973]: Warning: Permanently added 'smithi129,172.21.15.129' (ECDSA) to the list of known hosts. 2023-07-29T16:58:49.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:49 smithi078 bash[23704]: audit 2023-07-29T16:58:48.148217+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:49.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:49 smithi078 bash[23704]: audit 2023-07-29T16:58:48.441903+0000 mgr.y (mgr.14144) 5 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:58:49.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:49 smithi078 bash[23704]: audit 2023-07-29T16:58:48.570469+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:49.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:49 smithi078 bash[23704]: audit 2023-07-29T16:58:48.573724+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:49.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:49 smithi078 bash[23704]: audit 2023-07-29T16:58:48.581425+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:49.248 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi129' 2023-07-29T16:58:49.905 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch host ls --format=json 2023-07-29T16:58:50.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:50 smithi078 bash[23704]: audit 2023-07-29T16:58:49.239718+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:50.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:50 smithi078 bash[23704]: cephadm 2023-07-29T16:58:49.240432+0000 mgr.y (mgr.14144) 6 : cephadm [INF] Added host smithi129 2023-07-29T16:58:50.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:50 smithi078 bash[23704]: audit 2023-07-29T16:58:49.241027+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:58:50.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:50 smithi078 bash[23704]: audit 2023-07-29T16:58:49.483391+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:50.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:50 smithi078 bash[23704]: audit 2023-07-29T16:58:50.197665+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T16:58:53.426 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "smithi078", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "smithi129", "hostname": "smithi129", "labels": [], "status": ""}] 2023-07-29T16:58:54.106 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-29T16:58:54.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd crush tunables default 2023-07-29T16:58:54.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:54 smithi078 bash[23704]: audit 2023-07-29T16:58:53.416552+0000 mgr.y (mgr.14144) 7 : audit [DBG] from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:58:56.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:55 smithi078 bash[23704]: audit 2023-07-29T16:58:54.889783+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:56.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:55 smithi078 bash[23704]: cluster 2023-07-29T16:58:55.036278+0000 mgr.y (mgr.14144) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:58:56.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:55 smithi078 bash[23704]: audit 2023-07-29T16:58:55.167702+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:56.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:55 smithi078 bash[23704]: audit 2023-07-29T16:58:55.174971+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:58:58.065 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2023-07-29T16:58:58.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:58 smithi078 bash[23704]: cluster 2023-07-29T16:58:57.036593+0000 mgr.y (mgr.14144) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:58:58.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:58 smithi078 bash[23704]: audit 2023-07-29T16:58:57.123556+0000 mon.a (mon.0) 101 : audit [INF] from='client.? 172.21.15.78:0/1112357276' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-29T16:58:58.729 INFO:tasks.cephadm:Adding mon.a on smithi078 2023-07-29T16:58:58.729 INFO:tasks.cephadm:Adding mon.c on smithi078 2023-07-29T16:58:58.730 INFO:tasks.cephadm:Adding mon.b on smithi129 2023-07-29T16:58:58.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi129:172.21.15.129=b' 2023-07-29T16:58:59.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:59 smithi078 bash[23704]: audit 2023-07-29T16:58:58.053092+0000 mon.a (mon.0) 102 : audit [INF] from='client.? 172.21.15.78:0/1112357276' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-29T16:58:59.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:58:59 smithi078 bash[23704]: cluster 2023-07-29T16:58:58.053227+0000 mon.a (mon.0) 103 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:00.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:00 smithi078 bash[23704]: cluster 2023-07-29T16:58:59.037000+0000 mgr.y (mgr.14144) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:01.707 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mon update... 2023-07-29T16:59:02.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:02 smithi078 bash[23704]: cluster 2023-07-29T16:59:01.037440+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:02.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:02 smithi078 bash[23704]: audit 2023-07-29T16:59:01.699208+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:02.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:02 smithi078 bash[23704]: audit 2023-07-29T16:59:01.700703+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:02.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:02 smithi078 bash[23704]: audit 2023-07-29T16:59:01.711170+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:02.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:02 smithi078 bash[23704]: audit 2023-07-29T16:59:01.713452+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:02.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:02 smithi078 bash[23704]: audit 2023-07-29T16:59:01.715057+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:02.634 DEBUG:teuthology.orchestra.run.smithi078:mon.c> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.c.service 2023-07-29T16:59:02.637 DEBUG:teuthology.orchestra.run.smithi129:mon.b> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.b.service 2023-07-29T16:59:02.641 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-29T16:59:02.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph mon dump -f json 2023-07-29T16:59:02.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T16:59:03.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:03 smithi078 bash[23704]: audit 2023-07-29T16:59:01.690953+0000 mgr.y (mgr.14144) 12 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi129:172.21.15.129=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:03.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:03 smithi078 bash[23704]: cephadm 2023-07-29T16:59:01.694303+0000 mgr.y (mgr.14144) 13 : cephadm [INF] Saving service mon spec with placement smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi129:172.21.15.129=b;count:3 2023-07-29T16:59:03.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:03 smithi078 bash[23704]: cephadm 2023-07-29T16:59:01.716668+0000 mgr.y (mgr.14144) 14 : cephadm [INF] Deploying daemon mon.b on smithi129 2023-07-29T16:59:03.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T16:59:04.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:04 smithi078 bash[23704]: cluster 2023-07-29T16:59:03.037875+0000 mgr.y (mgr.14144) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:06.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:06 smithi078 bash[23704]: cluster 2023-07-29T16:59:05.038293+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:08.027 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-07-29T16:59:08.028 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"fsid":"eb1107b2-2e30-11ee-9b36-001a4aab830c","modified":"2023-07-29T16:57:35.076444Z","created":"2023-07-29T16:57:35.076444Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-29T16:59:08.030 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 1 2023-07-29T16:59:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:08 smithi078 bash[23704]: cluster 2023-07-29T16:59:07.038716+0000 mgr.y (mgr.14144) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:08 smithi078 bash[23704]: audit 2023-07-29T16:59:08.019069+0000 mon.a (mon.0) 109 : audit [DBG] from='client.? 172.21.15.129:0/430157898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:59:08.694 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:08 smithi129 systemd[1]: Started Ceph mon.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T16:59:09.765 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-29T16:59:09.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph mon dump -f json 2023-07-29T16:59:09.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.649+0000 7f1e12c9b700 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.649+0000 7f1e12c9b700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 8 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.649+0000 7f1e12c9b700 0 pidfile_write: ignore empty --pid-file 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 0 load: jerasure load: lrc load: isa 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: RocksDB version: 6.8.1 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Compile date Mar 30 2021 2023-07-29T16:59:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: DB SUMMARY 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: CURRENT file: CURRENT 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: IDENTITY file: IDENTITY 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 511 ; 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.error_if_exists: 0 2023-07-29T16:59:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.create_if_missing: 0 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.paranoid_checks: 1 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.env: 0x561257f741c0 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.fs: Posix File System 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.info_log: 0x561259998fe0 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.statistics: (nil) 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.use_fsync: 0 2023-07-29T16:59:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_log_file_size: 0 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.allow_fallocate: 1 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-29T16:59:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.use_direct_reads: 0 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.create_missing_column_families: 0 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.db_log_dir: 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_subcompactions: 1 2023-07-29T16:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_background_flushes: -1 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.advise_random_on_open: 1 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-29T16:59:09.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.write_buffer_manager: 0x561259a2fe90 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.rate_limiter: (nil) 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-29T16:59:09.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.unordered_write: 0 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T16:59:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.row_cache: None 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.wal_filter: None 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.preserve_deletes: 0 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.two_write_queues: 0 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.manual_wal_flush: 0 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.atomic_flush: 0 2023-07-29T16:59:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.log_readahead_size: 0 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_background_jobs: 2 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_background_compactions: -1 2023-07-29T16:59:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_total_wal_size: 0 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T16:59:09.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.max_open_files: -1 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.bytes_per_sync: 0 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Compression algorithms supported: 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T16:59:09.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kZSTD supported: 0 2023-07-29T16:59:09.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kXpressCompression supported: 0 2023-07-29T16:59:09.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-29T16:59:09.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kLZ4Compression supported: 1 2023-07-29T16:59:09.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kBZip2Compression supported: 0 2023-07-29T16:59:09.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kZlibCompression supported: 1 2023-07-29T16:59:09.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: kSnappyCompression supported: 1 2023-07-29T16:59:09.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.653+0000 7f1e12c9b700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T16:59:09.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2023-07-29T16:59:09.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-07-29T16:59:09.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T16:59:09.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.merge_operator: 2023-07-29T16:59:09.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_filter: None 2023-07-29T16:59:09.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_filter_factory: None 2023-07-29T16:59:09.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T16:59:09.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T16:59:09.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56125993c0d8) 2023-07-29T16:59:09.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: cache_index_and_filter_blocks: 1 2023-07-29T16:59:09.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T16:59:09.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T16:59:09.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: pin_top_level_index_and_filter: 1 2023-07-29T16:59:09.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: index_type: 0 2023-07-29T16:59:09.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: data_block_index_type: 0 2023-07-29T16:59:09.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: index_shortening: 1 2023-07-29T16:59:09.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: data_block_hash_table_util_ratio: 0.750000 2023-07-29T16:59:09.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: hash_index_allow_collision: 1 2023-07-29T16:59:09.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: checksum: 1 2023-07-29T16:59:09.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: no_block_cache: 0 2023-07-29T16:59:09.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_cache: 0x561259974f10 2023-07-29T16:59:09.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_cache_name: BinnedLRUCache 2023-07-29T16:59:09.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_cache_options: 2023-07-29T16:59:09.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: capacity : 536870912 2023-07-29T16:59:09.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: num_shard_bits : 4 2023-07-29T16:59:09.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: strict_capacity_limit : 0 2023-07-29T16:59:09.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: high_pri_pool_ratio: 0.000 2023-07-29T16:59:09.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_cache_compressed: (nil) 2023-07-29T16:59:09.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: persistent_cache: (nil) 2023-07-29T16:59:09.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_size: 4096 2023-07-29T16:59:09.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_size_deviation: 10 2023-07-29T16:59:09.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_restart_interval: 16 2023-07-29T16:59:09.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: index_block_restart_interval: 1 2023-07-29T16:59:09.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: metadata_block_size: 4096 2023-07-29T16:59:09.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: partition_filters: 0 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: use_delta_encoding: 1 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: filter_policy: rocksdb.BuiltinBloomFilter 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: whole_key_filtering: 1 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: verify_compression: 0 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: read_amp_bytes_per_bit: 0 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: format_version: 2 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: enable_index_compression: 1 2023-07-29T16:59:09.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: block_align: 0 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression: NoCompression 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T16:59:09.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.num_levels: 7 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T16:59:09.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T16:59:09.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:09 smithi078 bash[23704]: audit 2023-07-29T16:59:08.656475+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:09.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:09 smithi078 bash[23704]: audit 2023-07-29T16:59:08.657822+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:09.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:09 smithi078 bash[23704]: audit 2023-07-29T16:59:08.659661+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:09.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:09 smithi078 bash[23704]: cephadm 2023-07-29T16:59:08.661660+0000 mgr.y (mgr.14144) 18 : cephadm [INF] Deploying daemon mon.c on smithi078 2023-07-29T16:59:09.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:09 smithi078 bash[23704]: cluster 2023-07-29T16:59:09.039211+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:09.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:59:09.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T16:59:09.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T16:59:09.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression_opts.level: 32767 2023-07-29T16:59:09.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compression_opts.enabled: false 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T16:59:09.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T16:59:09.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.arena_block_size: 4194304 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T16:59:09.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T16:59:09.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.table_properties_collectors: 2023-07-29T16:59:09.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.inplace_update_support: 0 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.bloom_locality: 0 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.max_successive_merges: 0 2023-07-29T16:59:09.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T16:59:09.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-29T16:59:09.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.force_consistency_checks: 0 2023-07-29T16:59:09.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-29T16:59:09.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.ttl: 2592000 2023-07-29T16:59:09.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T16:59:09.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690649949660266, "job": 1, "event": "recovery_started", "log_files": [3]} 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690649949661100, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1690649949, "oldest_key_time": 3, "file_creation_time": 0}} 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2023-07-29T16:59:09.951 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690649949661725, "job": 1, "event": "recovery_finished"} 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.657+0000 7f1e12c9b700 4 rocksdb: DB pointer 0x561259a39800 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.661+0000 7f1df9ab5700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.661+0000 7f1df9ab5700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** DB Stats ** 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T16:59:09.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:59:09.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:59:09.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T16:59:09.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-07-29T16:59:09.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:59:09.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** Compaction Stats [default] ** 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** Compaction Stats [default] ** 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:59:09.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:59:09.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:59:09.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:59:09.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Cumulative compaction: 0.00 GB write, 0.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Interval compaction: 0.00 GB write, 0.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:59:09.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:59:09.957 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** Compaction Stats [default] ** 2023-07-29T16:59:09.957 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:59:09.957 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:59:09.957 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.957 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.957 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-29T16:59:09.958 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** Compaction Stats [default] ** 2023-07-29T16:59:09.958 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-29T16:59:09.958 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:59:09.958 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:59:09.959 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Cumulative compaction: 0.00 GB write, 0.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:59:09.960 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:59:09.960 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:59:09.960 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:59:09.960 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.661+0000 7f1e12c9b700 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-29T16:59:09.960 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.661+0000 7f1e12c9b700 0 using public_addr v2:172.21.15.129:0/0 -> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2023-07-29T16:59:09.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.661+0000 7f1e12c9b700 0 starting mon.b rank -1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:59:09.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.661+0000 7f1e12c9b700 1 mon.b@-1(???) e0 preinit fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:59:09.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.689+0000 7f1dfc2ba700 0 mon.b@-1(synchronizing).mds e1 new map 2023-07-29T16:59:09.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.689+0000 7f1dfc2ba700 0 mon.b@-1(synchronizing).mds e1 print_map 2023-07-29T16:59:09.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: e1 2023-07-29T16:59:09.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: legacy client fscid: -1 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: No filesystems configured 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.689+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.689+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.689+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-29T16:59:09.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: audit 2023-07-29T16:59:08.656475+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:09.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: audit 2023-07-29T16:59:08.657822+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: audit 2023-07-29T16:59:08.659661+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: cephadm 2023-07-29T16:59:08.661660+0000 mgr.y (mgr.14144) 18 : cephadm [INF] Deploying daemon mon.c on smithi078 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: cluster 2023-07-29T16:59:09.039211+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 1 mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expand map: {default=false} 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta from 'false' to 'false' 2023-07-29T16:59:09.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expanded map: {default=false} 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expand map: {default=info} 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta from 'info' to 'info' 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expanded map: {default=info} 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expand map: {default=daemon} 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expanded map: {default=daemon} 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expand map: {default=debug} 2023-07-29T16:59:09.965 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta from 'debug' to 'debug' 2023-07-29T16:59:09.966 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 20 expand_channel_meta expanded map: {default=debug} 2023-07-29T16:59:09.966 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:09 smithi129 bash[23526]: debug 2023-07-29T16:59:09.693+0000 7f1dfc2ba700 10 mon.b@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-29T16:59:13.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:13 smithi078 bash[28259]: debug 2023-07-29T16:59:13.599+0000 7f82efa4e700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2023-07-29T16:59:15.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:09.706876+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:59:15.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:09.707253+0000 mon.a (mon.0) 115 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:15.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:09.709971+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:10.701621+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:11.039589+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:15.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:11.701783+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:11.704667+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:15.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:12.701831+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:13.039930+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:13.609605+0000 mon.a (mon.0) 120 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:13.701996+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:14.609791+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:14.701951+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:14.732537+0000 mon.a (mon.0) 124 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-29T16:59:15.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:14.738535+0000 mon.a (mon.0) 125 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} 2023-07-29T16:59:15.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:14.738646+0000 mon.a (mon.0) 126 : cluster [DBG] fsmap 2023-07-29T16:59:15.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:14.738705+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:15.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:14.739084+0000 mon.a (mon.0) 128 : cluster [DBG] mgrmap e13: y(active, since 39s) 2023-07-29T16:59:15.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: cluster 2023-07-29T16:59:14.746032+0000 mon.a (mon.0) 129 : cluster [INF] overall HEALTH_OK 2023-07-29T16:59:15.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:14.750485+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:15.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:14 smithi129 bash[23526]: audit 2023-07-29T16:59:14.751656+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:15.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:09.706876+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:59:15.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:09.707253+0000 mon.a (mon.0) 115 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:15.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:09.709971+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:10.701621+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:11.039589+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:11.701783+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:11.704667+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:12.701831+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:13.039930+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:15.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:13.609605+0000 mon.a (mon.0) 120 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:13.701996+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:14.609791+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:14.701951+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:14.732537+0000 mon.a (mon.0) 124 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:14.738535+0000 mon.a (mon.0) 125 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:14.738646+0000 mon.a (mon.0) 126 : cluster [DBG] fsmap 2023-07-29T16:59:15.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:14.738705+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:15.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:14.739084+0000 mon.a (mon.0) 128 : cluster [DBG] mgrmap e13: y(active, since 39s) 2023-07-29T16:59:15.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: cluster 2023-07-29T16:59:14.746032+0000 mon.a (mon.0) 129 : cluster [INF] overall HEALTH_OK 2023-07-29T16:59:15.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:14.750485+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:15.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:14 smithi078 bash[23704]: audit 2023-07-29T16:59:14.751656+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:20.635 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-07-29T16:59:20.635 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"fsid":"eb1107b2-2e30-11ee-9b36-001a4aab830c","modified":"2023-07-29T16:59:15.611149Z","created":"2023-07-29T16:57:35.076444Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3301","nonce":0},{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-29T16:59:20.640 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 3 2023-07-29T16:59:21.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:15.617029+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:59:21.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:15.617548+0000 mon.a (mon.0) 134 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:21.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:15.619469+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:15.620348+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:15.620821+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:15.831917+0000 mon.a (mon.0) 137 : audit [DBG] from='client.? 172.21.15.129:0/3836333105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:16.610669+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:17.040739+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:17.610506+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:18.610892+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:19.041211+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:19.610675+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.610808+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.619953+0000 mon.a (mon.0) 143 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.625363+0000 mon.a (mon.0) 144 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.625463+0000 mon.a (mon.0) 145 : cluster [DBG] fsmap 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.625521+0000 mon.a (mon.0) 146 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:21.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.625836+0000 mon.a (mon.0) 147 : cluster [DBG] mgrmap e13: y(active, since 45s) 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.626173+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.633792+0000 mon.a (mon.0) 149 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.633840+0000 mon.a (mon.0) 150 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: cluster 2023-07-29T16:59:20.633871+0000 mon.a (mon.0) 151 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] is down (out of quorum) 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.640168+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.644745+0000 mon.a (mon.0) 153 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:21.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.651480+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:21.166 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.654003+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:21.166 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.655591+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:21.166 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:20 smithi129 bash[23526]: audit 2023-07-29T16:59:20.657086+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:21.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:15.617029+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:15.617548+0000 mon.a (mon.0) 134 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:15.619469+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:15.620348+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:15.620821+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:15.831917+0000 mon.a (mon.0) 137 : audit [DBG] from='client.? 172.21.15.129:0/3836333105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:16.610669+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:17.040739+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:17.610506+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:18.610892+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:19.041211+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:19.610675+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.610808+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.619953+0000 mon.a (mon.0) 143 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-29T16:59:21.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.625363+0000 mon.a (mon.0) 144 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} 2023-07-29T16:59:21.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.625463+0000 mon.a (mon.0) 145 : cluster [DBG] fsmap 2023-07-29T16:59:21.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.625521+0000 mon.a (mon.0) 146 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:21.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.625836+0000 mon.a (mon.0) 147 : cluster [DBG] mgrmap e13: y(active, since 45s) 2023-07-29T16:59:21.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.626173+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.633792+0000 mon.a (mon.0) 149 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.633840+0000 mon.a (mon.0) 150 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: cluster 2023-07-29T16:59:20.633871+0000 mon.a (mon.0) 151 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] is down (out of quorum) 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.640168+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.644745+0000 mon.a (mon.0) 153 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.651480+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:21.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.654003+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:21.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.655591+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:21.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:20 smithi078 bash[23704]: audit 2023-07-29T16:59:20.657086+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:21.392 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-29T16:59:21.392 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph config generate-minimal-conf 2023-07-29T16:59:22.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[23704]: cephadm 2023-07-29T16:59:20.653467+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-07-29T16:59:22.397 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[23704]: cephadm 2023-07-29T16:59:20.658360+0000 mgr.y (mgr.14144) 26 : cephadm [INF] Reconfiguring daemon mon.a on smithi078 2023-07-29T16:59:22.397 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[23704]: cluster 2023-07-29T16:59:21.041662+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:22.397 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[23704]: audit 2023-07-29T16:59:21.611063+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:22.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:22 smithi129 bash[23526]: cephadm 2023-07-29T16:59:20.653467+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-07-29T16:59:22.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:22 smithi129 bash[23526]: cephadm 2023-07-29T16:59:20.658360+0000 mgr.y (mgr.14144) 26 : cephadm [INF] Reconfiguring daemon mon.a on smithi078 2023-07-29T16:59:22.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:22 smithi129 bash[23526]: cluster 2023-07-29T16:59:21.041662+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:22.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:22 smithi129 bash[23526]: audit 2023-07-29T16:59:21.611063+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:22.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[28259]: cephadm 2023-07-29T16:59:20.653467+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-07-29T16:59:22.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[28259]: cephadm 2023-07-29T16:59:20.658360+0000 mgr.y (mgr.14144) 26 : cephadm [INF] Reconfiguring daemon mon.a on smithi078 2023-07-29T16:59:22.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[28259]: cluster 2023-07-29T16:59:21.041662+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:22.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:22 smithi078 bash[28259]: audit 2023-07-29T16:59:21.611063+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:23.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:17.612773+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2023-07-29T16:59:23.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.625397+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2023-07-29T16:59:23.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.626128+0000 mon.a (mon.0) 160 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:23.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.626948+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:23.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.649449+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T16:59:23.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.655719+0000 mon.a (mon.0) 162 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} 2023-07-29T16:59:23.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.655831+0000 mon.a (mon.0) 163 : cluster [DBG] fsmap 2023-07-29T16:59:23.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.655891+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:23.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.656209+0000 mon.a (mon.0) 165 : cluster [DBG] mgrmap e13: y(active, since 47s) 2023-07-29T16:59:23.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.656521+0000 mon.a (mon.0) 166 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T16:59:23.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.656571+0000 mon.a (mon.0) 167 : cluster [INF] Cluster is now healthy 2023-07-29T16:59:23.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[23704]: cluster 2023-07-29T16:59:22.662784+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2023-07-29T16:59:23.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:17.612773+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2023-07-29T16:59:23.336 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.625397+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2023-07-29T16:59:23.336 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.626128+0000 mon.a (mon.0) 160 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:23.336 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.626948+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:23.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.649449+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T16:59:23.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.655719+0000 mon.a (mon.0) 162 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} 2023-07-29T16:59:23.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.655831+0000 mon.a (mon.0) 163 : cluster [DBG] fsmap 2023-07-29T16:59:23.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.655891+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:23.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.656209+0000 mon.a (mon.0) 165 : cluster [DBG] mgrmap e13: y(active, since 47s) 2023-07-29T16:59:23.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.656521+0000 mon.a (mon.0) 166 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T16:59:23.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.656571+0000 mon.a (mon.0) 167 : cluster [INF] Cluster is now healthy 2023-07-29T16:59:23.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:23 smithi078 bash[28259]: cluster 2023-07-29T16:59:22.662784+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2023-07-29T16:59:23.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:17.612773+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2023-07-29T16:59:23.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.625397+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2023-07-29T16:59:23.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.626128+0000 mon.a (mon.0) 160 : cluster [INF] mon.a calling monitor election 2023-07-29T16:59:23.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.626948+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2023-07-29T16:59:23.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.649449+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T16:59:23.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.655719+0000 mon.a (mon.0) 162 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} 2023-07-29T16:59:23.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.655831+0000 mon.a (mon.0) 163 : cluster [DBG] fsmap 2023-07-29T16:59:23.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.655891+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:59:23.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.656209+0000 mon.a (mon.0) 165 : cluster [DBG] mgrmap e13: y(active, since 47s) 2023-07-29T16:59:23.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.656521+0000 mon.a (mon.0) 166 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T16:59:23.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.656571+0000 mon.a (mon.0) 167 : cluster [INF] Cluster is now healthy 2023-07-29T16:59:23.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:23 smithi129 bash[23526]: cluster 2023-07-29T16:59:22.662784+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2023-07-29T16:59:24.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:24 smithi129 bash[23526]: cluster 2023-07-29T16:59:23.042112+0000 mgr.y (mgr.14144) 28 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:24.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:24 smithi129 bash[23526]: audit 2023-07-29T16:59:23.365381+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:24.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:24 smithi129 bash[23526]: audit 2023-07-29T16:59:23.366860+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:24.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:24 smithi129 bash[23526]: audit 2023-07-29T16:59:23.368502+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:24.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:24 smithi129 bash[23526]: audit 2023-07-29T16:59:23.370049+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:24.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:24 smithi129 bash[23526]: audit 2023-07-29T16:59:23.611186+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:24.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[23704]: cluster 2023-07-29T16:59:23.042112+0000 mgr.y (mgr.14144) 28 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:24.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[23704]: audit 2023-07-29T16:59:23.365381+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:24.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[23704]: audit 2023-07-29T16:59:23.366860+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:24.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[23704]: audit 2023-07-29T16:59:23.368502+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:24.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[23704]: audit 2023-07-29T16:59:23.370049+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:24.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[23704]: audit 2023-07-29T16:59:23.611186+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:24.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[28259]: cluster 2023-07-29T16:59:23.042112+0000 mgr.y (mgr.14144) 28 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:24.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[28259]: audit 2023-07-29T16:59:23.365381+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:24.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[28259]: audit 2023-07-29T16:59:23.366860+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:24.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[28259]: audit 2023-07-29T16:59:23.368502+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:24.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[28259]: audit 2023-07-29T16:59:23.370049+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:24.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:24 smithi078 bash[28259]: audit 2023-07-29T16:59:23.611186+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:59:25.321 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:25 smithi078 bash[28259]: cephadm 2023-07-29T16:59:23.366348+0000 mgr.y (mgr.14144) 29 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-07-29T16:59:25.321 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:25 smithi078 bash[28259]: cephadm 2023-07-29T16:59:23.371685+0000 mgr.y (mgr.14144) 30 : cephadm [INF] Reconfiguring daemon mon.c on smithi078 2023-07-29T16:59:25.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:25 smithi078 bash[23704]: cephadm 2023-07-29T16:59:23.366348+0000 mgr.y (mgr.14144) 29 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-07-29T16:59:25.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:25 smithi078 bash[23704]: cephadm 2023-07-29T16:59:23.371685+0000 mgr.y (mgr.14144) 30 : cephadm [INF] Reconfiguring daemon mon.c on smithi078 2023-07-29T16:59:25.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:25 smithi129 bash[23526]: cephadm 2023-07-29T16:59:23.366348+0000 mgr.y (mgr.14144) 29 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-07-29T16:59:25.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:25 smithi129 bash[23526]: cephadm 2023-07-29T16:59:23.371685+0000 mgr.y (mgr.14144) 30 : cephadm [INF] Reconfiguring daemon mon.c on smithi078 2023-07-29T16:59:25.950 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:59:25.950 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2023-07-29T16:59:25.950 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T16:59:25.951 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] 2023-07-29T16:59:26.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:26 smithi129 bash[23526]: cluster 2023-07-29T16:59:25.042643+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:26.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:26 smithi129 bash[23526]: audit 2023-07-29T16:59:25.867763+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:26.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:26 smithi129 bash[23526]: audit 2023-07-29T16:59:25.868855+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:26.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:26 smithi129 bash[23526]: audit 2023-07-29T16:59:25.870219+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:26.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:26 smithi129 bash[23526]: audit 2023-07-29T16:59:25.871403+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:26.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:26 smithi129 bash[23526]: audit 2023-07-29T16:59:25.942049+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? 172.21.15.78:0/1041944801' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:26.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[23704]: cluster 2023-07-29T16:59:25.042643+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:26.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[23704]: audit 2023-07-29T16:59:25.867763+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:26.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[23704]: audit 2023-07-29T16:59:25.868855+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:26.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[23704]: audit 2023-07-29T16:59:25.870219+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:26.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[23704]: audit 2023-07-29T16:59:25.871403+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:26.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[23704]: audit 2023-07-29T16:59:25.942049+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? 172.21.15.78:0/1041944801' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:26.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[28259]: cluster 2023-07-29T16:59:25.042643+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:26.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[28259]: audit 2023-07-29T16:59:25.867763+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:26.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[28259]: audit 2023-07-29T16:59:25.868855+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:59:26.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[28259]: audit 2023-07-29T16:59:25.870219+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:59:26.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[28259]: audit 2023-07-29T16:59:25.871403+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:26.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:26 smithi078 bash[28259]: audit 2023-07-29T16:59:25.942049+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? 172.21.15.78:0/1041944801' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:26.520 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-29T16:59:26.520 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:59:26.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2023-07-29T16:59:26.534 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:59:26.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:59:26.547 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:59:26.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2023-07-29T16:59:26.565 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:59:26.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:59:26.621 INFO:tasks.cephadm:Adding mgr.y on smithi078 2023-07-29T16:59:26.622 INFO:tasks.cephadm:Adding mgr.x on smithi129 2023-07-29T16:59:26.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch apply mgr '2;smithi078=y;smithi129=x' 2023-07-29T16:59:27.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:27 smithi129 bash[23526]: cephadm 2023-07-29T16:59:25.868383+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-07-29T16:59:27.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:27 smithi129 bash[23526]: cephadm 2023-07-29T16:59:25.872465+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Reconfiguring daemon mon.b on smithi129 2023-07-29T16:59:27.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:27 smithi078 bash[23704]: cephadm 2023-07-29T16:59:25.868383+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-07-29T16:59:27.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:27 smithi078 bash[23704]: cephadm 2023-07-29T16:59:25.872465+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Reconfiguring daemon mon.b on smithi129 2023-07-29T16:59:27.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:27 smithi078 bash[28259]: cephadm 2023-07-29T16:59:25.868383+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-07-29T16:59:27.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:27 smithi078 bash[28259]: cephadm 2023-07-29T16:59:25.872465+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Reconfiguring daemon mon.b on smithi129 2023-07-29T16:59:28.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:28 smithi129 bash[23526]: cluster 2023-07-29T16:59:27.043125+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:28.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:28 smithi078 bash[23704]: cluster 2023-07-29T16:59:27.043125+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:28.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:28 smithi078 bash[28259]: cluster 2023-07-29T16:59:27.043125+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:29.859 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:29 smithi129 bash[23526]: audit 2023-07-29T16:59:28.463958+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:29.859 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:29 smithi129 bash[23526]: audit 2023-07-29T16:59:28.465659+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:29.859 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:29 smithi129 bash[23526]: audit 2023-07-29T16:59:28.475834+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:29.859 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:29 smithi129 bash[23526]: cluster 2023-07-29T16:59:29.043609+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:29.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[23704]: audit 2023-07-29T16:59:28.463958+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:29.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[23704]: audit 2023-07-29T16:59:28.465659+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:29.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[23704]: audit 2023-07-29T16:59:28.475834+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:29.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[23704]: cluster 2023-07-29T16:59:29.043609+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:29.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[28259]: audit 2023-07-29T16:59:28.463958+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:29.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[28259]: audit 2023-07-29T16:59:28.465659+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:29.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[28259]: audit 2023-07-29T16:59:28.475834+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:29.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:29 smithi078 bash[28259]: cluster 2023-07-29T16:59:29.043609+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:30.270 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mgr update... 2023-07-29T16:59:31.215 DEBUG:teuthology.orchestra.run.smithi129:mgr.x> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x.service 2023-07-29T16:59:31.217 INFO:tasks.cephadm:Deploying OSDs... 2023-07-29T16:59:31.218 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T16:59:31.218 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2023-07-29T16:59:31.225 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-29T16:59:31.225 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2023-07-29T16:59:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-29T16:59:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.275 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-07-29T16:59:31.275 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.276 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-29 16:55:09.456278408 +0000 2023-07-29T16:59:31.276 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-29 16:55:09.452278501 +0000 2023-07-29T16:59:31.276 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-29 16:55:09.452278501 +0000 2023-07-29T16:59:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-29T16:59:31.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-29T16:59:31.288 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-29T16:59:31.288 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-29T16:59:31.288 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00211335 s, 242 kB/s 2023-07-29T16:59:31.289 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-29T16:59:31.296 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2023-07-29T16:59:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-29T16:59:31.301 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.301 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-07-29T16:59:31.301 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.302 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-29 16:55:10.252260094 +0000 2023-07-29T16:59:31.302 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-29 16:55:09.972266537 +0000 2023-07-29T16:59:31.302 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-29 16:55:09.972266537 +0000 2023-07-29T16:59:31.302 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-29T16:59:31.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-29T16:59:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-29T16:59:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-29T16:59:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000395793 s, 1.3 MB/s 2023-07-29T16:59:31.315 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-29T16:59:31.370 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2023-07-29T16:59:31.378 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-29T16:59:31.379 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.379 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-07-29T16:59:31.379 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.379 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-29 16:55:10.460255310 +0000 2023-07-29T16:59:31.380 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-29 16:55:10.456255401 +0000 2023-07-29T16:59:31.380 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-29 16:55:10.456255401 +0000 2023-07-29T16:59:31.380 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-29T16:59:31.381 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-29T16:59:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-29T16:59:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-29T16:59:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000396989 s, 1.3 MB/s 2023-07-29T16:59:31.400 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-29T16:59:31.408 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2023-07-29T16:59:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-29T16:59:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-07-29T16:59:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-29 16:55:11.264236813 +0000 2023-07-29T16:59:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-29 16:55:10.980243346 +0000 2023-07-29T16:59:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-29 16:55:10.980243346 +0000 2023-07-29T16:59:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-29T16:59:31.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-29T16:59:31.477 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-29T16:59:31.477 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-29T16:59:31.477 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000410161 s, 1.2 MB/s 2023-07-29T16:59:31.478 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-29T16:59:31.531 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T16:59:31.531 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2023-07-29T16:59:31.538 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-29T16:59:31.539 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2023-07-29T16:59:31.588 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-29T16:59:31.588 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.588 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 357 Links: 1 2023-07-29T16:59:31.588 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.588 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-07-29 16:55:09.502924266 +0000 2023-07-29T16:59:31.589 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-07-29 16:55:09.194931365 +0000 2023-07-29T16:59:31.589 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-07-29 16:55:09.194931365 +0000 2023-07-29T16:59:31.589 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-07-29T16:59:31.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-29T16:59:31.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.253540+0000 mgr.y (mgr.14144) 36 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi078=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:31.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: cephadm 2023-07-29T16:59:30.255088+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Saving service mgr spec with placement smithi078=y;smithi129=x;count:2 2023-07-29T16:59:31.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.261481+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:31.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.262923+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:31.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.272729+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:31.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.274338+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:59:31.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.279911+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:59:31.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.281231+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:59:31.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: audit 2023-07-29T16:59:30.282816+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:31.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: cephadm 2023-07-29T16:59:30.284219+0000 mgr.y (mgr.14144) 38 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T16:59:31.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:31 smithi129 bash[23526]: cluster 2023-07-29T16:59:31.044033+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:31.604 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-07-29T16:59:31.604 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-07-29T16:59:31.604 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000448546 s, 1.1 MB/s 2023-07-29T16:59:31.605 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-29T16:59:31.613 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2023-07-29T16:59:31.664 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-29T16:59:31.665 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.665 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-07-29T16:59:31.665 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.665 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-07-29 16:55:10.022912280 +0000 2023-07-29T16:59:31.665 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-07-29 16:55:09.722919195 +0000 2023-07-29T16:59:31.665 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-07-29 16:55:09.722919195 +0000 2023-07-29T16:59:31.666 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-07-29T16:59:31.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-29T16:59:31.676 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-07-29T16:59:31.677 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-07-29T16:59:31.677 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000398188 s, 1.3 MB/s 2023-07-29T16:59:31.678 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-29T16:59:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.253540+0000 mgr.y (mgr.14144) 36 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi078=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: cephadm 2023-07-29T16:59:30.255088+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Saving service mgr spec with placement smithi078=y;smithi129=x;count:2 2023-07-29T16:59:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.261481+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.262923+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.272729+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.274338+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:59:31.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.279911+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:59:31.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.281231+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:59:31.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: audit 2023-07-29T16:59:30.282816+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:31.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: cephadm 2023-07-29T16:59:30.284219+0000 mgr.y (mgr.14144) 38 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T16:59:31.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[23704]: cluster 2023-07-29T16:59:31.044033+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:31.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.253540+0000 mgr.y (mgr.14144) 36 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi078=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: cephadm 2023-07-29T16:59:30.255088+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Saving service mgr spec with placement smithi078=y;smithi129=x;count:2 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.261481+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.262923+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.272729+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.274338+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.279911+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:59:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.281231+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:59:31.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: audit 2023-07-29T16:59:30.282816+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:31.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: cephadm 2023-07-29T16:59:30.284219+0000 mgr.y (mgr.14144) 38 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T16:59:31.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:31 smithi078 bash[28259]: cluster 2023-07-29T16:59:31.044033+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:31.689 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2023-07-29T16:59:31.739 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-29T16:59:31.739 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.739 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-07-29T16:59:31.740 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.740 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-07-29 16:55:10.522900755 +0000 2023-07-29T16:59:31.740 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-07-29 16:55:10.250907025 +0000 2023-07-29T16:59:31.740 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-07-29 16:55:10.250907025 +0000 2023-07-29T16:59:31.740 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-07-29T16:59:31.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-29T16:59:31.752 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-07-29T16:59:31.752 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-07-29T16:59:31.753 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000298408 s, 1.7 MB/s 2023-07-29T16:59:31.753 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-29T16:59:31.762 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2023-07-29T16:59:31.810 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-29T16:59:31.810 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:59:31.810 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 375 Links: 1 2023-07-29T16:59:31.811 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:59:31.811 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-07-29 16:55:10.990889968 +0000 2023-07-29T16:59:31.811 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-07-29 16:55:10.714896329 +0000 2023-07-29T16:59:31.811 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-07-29 16:55:10.714896329 +0000 2023-07-29T16:59:31.811 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-07-29T16:59:31.812 INFO:journalctl@ceph.mgr.x.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T16:59:31.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-29T16:59:31.824 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-07-29T16:59:31.825 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-07-29T16:59:31.825 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000419398 s, 1.2 MB/s 2023-07-29T16:59:31.826 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-29T16:59:31.875 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2023-07-29T16:59:31.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-29T16:59:34.161 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:34 smithi129 bash[23526]: audit 2023-07-29T16:59:33.041661+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:34.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:34 smithi129 bash[23526]: audit 2023-07-29T16:59:33.043820+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:34.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:34 smithi129 bash[23526]: cluster 2023-07-29T16:59:33.044417+0000 mgr.y (mgr.14144) 40 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:34.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 16:59:34 smithi129 bash[24408]: debug 2023-07-29T16:59:34.017+0000 7f9e19e96700 1 -- 172.21.15.129:0/3643241947 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561432f5e4e0 con 0x56143304ec00 2023-07-29T16:59:34.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:34 smithi078 bash[23704]: audit 2023-07-29T16:59:33.041661+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:34.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:34 smithi078 bash[23704]: audit 2023-07-29T16:59:33.043820+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:34.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:34 smithi078 bash[23704]: cluster 2023-07-29T16:59:33.044417+0000 mgr.y (mgr.14144) 40 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:34.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:34 smithi078 bash[28259]: audit 2023-07-29T16:59:33.041661+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:34.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:34 smithi078 bash[28259]: audit 2023-07-29T16:59:33.043820+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:34.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:34 smithi078 bash[28259]: cluster 2023-07-29T16:59:33.044417+0000 mgr.y (mgr.14144) 40 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-29T16:59:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-29T16:59:35.210 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-29T16:59:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-29T16:59:35.210 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0220084 s, 476 MB/s 2023-07-29T16:59:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-29T16:59:35.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2023-07-29T16:59:36.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:36 smithi078 bash[23704]: cluster 2023-07-29T16:59:35.044735+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:36.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:36 smithi078 bash[23704]: audit 2023-07-29T16:59:35.239229+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:59:36.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:36 smithi078 bash[23704]: audit 2023-07-29T16:59:35.241666+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T16:59:36.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:36 smithi078 bash[28259]: cluster 2023-07-29T16:59:35.044735+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:36.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:36 smithi078 bash[28259]: audit 2023-07-29T16:59:35.239229+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:59:36.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:36 smithi078 bash[28259]: audit 2023-07-29T16:59:35.241666+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T16:59:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:36 smithi129 bash[23526]: cluster 2023-07-29T16:59:35.044735+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:36.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:36 smithi129 bash[23526]: audit 2023-07-29T16:59:35.239229+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:59:36.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:36 smithi129 bash[23526]: audit 2023-07-29T16:59:35.241666+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T16:59:37.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: audit 2023-07-29T16:59:36.068481+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:37.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: audit 2023-07-29T16:59:36.075333+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:37.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: cephadm 2023-07-29T16:59:36.077047+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-07-29T16:59:37.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: audit 2023-07-29T16:59:36.077484+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:59:37.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: audit 2023-07-29T16:59:36.078960+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:59:37.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: audit 2023-07-29T16:59:36.080245+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:37.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:37 smithi129 bash[23526]: cephadm 2023-07-29T16:59:36.081364+0000 mgr.y (mgr.14144) 43 : cephadm [INF] Reconfiguring daemon mgr.y on smithi078 2023-07-29T16:59:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: audit 2023-07-29T16:59:36.068481+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: audit 2023-07-29T16:59:36.075333+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: cephadm 2023-07-29T16:59:36.077047+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-07-29T16:59:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: audit 2023-07-29T16:59:36.077484+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:59:37.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: audit 2023-07-29T16:59:36.078960+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:59:37.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: audit 2023-07-29T16:59:36.080245+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:37.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[23704]: cephadm 2023-07-29T16:59:36.081364+0000 mgr.y (mgr.14144) 43 : cephadm [INF] Reconfiguring daemon mgr.y on smithi078 2023-07-29T16:59:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: audit 2023-07-29T16:59:36.068481+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: audit 2023-07-29T16:59:36.075333+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: cephadm 2023-07-29T16:59:36.077047+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-07-29T16:59:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: audit 2023-07-29T16:59:36.077484+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:59:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: audit 2023-07-29T16:59:36.078960+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:59:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: audit 2023-07-29T16:59:36.080245+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:37.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:37 smithi078 bash[28259]: cephadm 2023-07-29T16:59:36.081364+0000 mgr.y (mgr.14144) 43 : cephadm [INF] Reconfiguring daemon mgr.y on smithi078 2023-07-29T16:59:38.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:38 smithi129 bash[23526]: cluster 2023-07-29T16:59:37.045200+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:38.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:38 smithi078 bash[28259]: cluster 2023-07-29T16:59:37.045200+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:38.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:38 smithi078 bash[23704]: cluster 2023-07-29T16:59:37.045200+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:39.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[23704]: audit 2023-07-29T16:59:38.753854+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:39.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[23704]: audit 2023-07-29T16:59:38.754968+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:39.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[23704]: audit 2023-07-29T16:59:38.761421+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:39.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[23704]: cluster 2023-07-29T16:59:39.045762+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:39.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[28259]: audit 2023-07-29T16:59:38.753854+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:39.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[28259]: audit 2023-07-29T16:59:38.754968+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:39.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[28259]: audit 2023-07-29T16:59:38.761421+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:39.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:39 smithi078 bash[28259]: cluster 2023-07-29T16:59:39.045762+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:40.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:39 smithi129 bash[23526]: audit 2023-07-29T16:59:38.753854+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:40.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:39 smithi129 bash[23526]: audit 2023-07-29T16:59:38.754968+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:40.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:39 smithi129 bash[23526]: audit 2023-07-29T16:59:38.761421+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:40.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:39 smithi129 bash[23526]: cluster 2023-07-29T16:59:39.045762+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:41.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: cluster 2023-07-29T16:59:39.833521+0000 mon.a (mon.0) 200 : cluster [DBG] Standby manager daemon x started 2023-07-29T16:59:41.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:39.836465+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T16:59:41.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:39.837507+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:59:41.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:39.839023+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T16:59:41.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:39.839593+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:59:41.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:40.040738+0000 mgr.y (mgr.14144) 46 : audit [DBG] from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:41.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:40.042131+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:59:41.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: cephadm 2023-07-29T16:59:40.044553+0000 mgr.y (mgr.14144) 47 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T16:59:41.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:40.044904+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:59:41.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[28259]: audit 2023-07-29T16:59:40.045996+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:41.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: cluster 2023-07-29T16:59:39.833521+0000 mon.a (mon.0) 200 : cluster [DBG] Standby manager daemon x started 2023-07-29T16:59:41.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:39.836465+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T16:59:41.061 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:39.837507+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:39.839023+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:39.839593+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:40.040738+0000 mgr.y (mgr.14144) 46 : audit [DBG] from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:40.042131+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: cephadm 2023-07-29T16:59:40.044553+0000 mgr.y (mgr.14144) 47 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:40.044904+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:59:41.062 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:40 smithi078 bash[23704]: audit 2023-07-29T16:59:40.045996+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:41.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: cluster 2023-07-29T16:59:39.833521+0000 mon.a (mon.0) 200 : cluster [DBG] Standby manager daemon x started 2023-07-29T16:59:41.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:39.836465+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T16:59:41.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:39.837507+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:59:41.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:39.839023+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T16:59:41.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:39.839593+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.129:0/3431575737' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:59:41.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:40.040738+0000 mgr.y (mgr.14144) 46 : audit [DBG] from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:59:41.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:40.042131+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:59:41.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: cephadm 2023-07-29T16:59:40.044553+0000 mgr.y (mgr.14144) 47 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T16:59:41.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:40.044904+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:59:41.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:40 smithi129 bash[23526]: audit 2023-07-29T16:59:40.045996+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:42.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:41 smithi129 bash[23526]: cluster 2023-07-29T16:59:40.778396+0000 mon.a (mon.0) 204 : cluster [DBG] mgrmap e14: y(active, since 65s), standbys: x 2023-07-29T16:59:42.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:41 smithi129 bash[23526]: audit 2023-07-29T16:59:40.778635+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T16:59:42.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:41 smithi129 bash[23526]: cluster 2023-07-29T16:59:41.046274+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:42.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:41 smithi078 bash[23704]: cluster 2023-07-29T16:59:40.778396+0000 mon.a (mon.0) 204 : cluster [DBG] mgrmap e14: y(active, since 65s), standbys: x 2023-07-29T16:59:42.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:41 smithi078 bash[23704]: audit 2023-07-29T16:59:40.778635+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T16:59:42.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:41 smithi078 bash[23704]: cluster 2023-07-29T16:59:41.046274+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:42.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:41 smithi078 bash[28259]: cluster 2023-07-29T16:59:40.778396+0000 mon.a (mon.0) 204 : cluster [DBG] mgrmap e14: y(active, since 65s), standbys: x 2023-07-29T16:59:42.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:41 smithi078 bash[28259]: audit 2023-07-29T16:59:40.778635+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T16:59:42.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:41 smithi078 bash[28259]: cluster 2023-07-29T16:59:41.046274+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:44.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:44 smithi129 bash[23526]: cluster 2023-07-29T16:59:43.046741+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:44.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:44 smithi129 bash[23526]: audit 2023-07-29T16:59:43.654699+0000 mon.c (mon.2) 4 : audit [INF] from='client.? 172.21.15.78:0/1780671315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]: dispatch 2023-07-29T16:59:44.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:44 smithi129 bash[23526]: audit 2023-07-29T16:59:43.655402+0000 mon.a (mon.0) 206 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]: dispatch 2023-07-29T16:59:44.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:44 smithi129 bash[23526]: audit 2023-07-29T16:59:43.662896+0000 mon.a (mon.0) 207 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]': finished 2023-07-29T16:59:44.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:44 smithi129 bash[23526]: cluster 2023-07-29T16:59:43.663098+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-07-29T16:59:44.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:44 smithi129 bash[23526]: audit 2023-07-29T16:59:43.663249+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:59:44.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[23704]: cluster 2023-07-29T16:59:43.046741+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:44.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[23704]: audit 2023-07-29T16:59:43.654699+0000 mon.c (mon.2) 4 : audit [INF] from='client.? 172.21.15.78:0/1780671315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]: dispatch 2023-07-29T16:59:44.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[23704]: audit 2023-07-29T16:59:43.655402+0000 mon.a (mon.0) 206 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]: dispatch 2023-07-29T16:59:44.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[23704]: audit 2023-07-29T16:59:43.662896+0000 mon.a (mon.0) 207 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]': finished 2023-07-29T16:59:44.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[23704]: cluster 2023-07-29T16:59:43.663098+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-07-29T16:59:44.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[23704]: audit 2023-07-29T16:59:43.663249+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:59:44.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[28259]: cluster 2023-07-29T16:59:43.046741+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:44.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[28259]: audit 2023-07-29T16:59:43.654699+0000 mon.c (mon.2) 4 : audit [INF] from='client.? 172.21.15.78:0/1780671315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]: dispatch 2023-07-29T16:59:44.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[28259]: audit 2023-07-29T16:59:43.655402+0000 mon.a (mon.0) 206 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]: dispatch 2023-07-29T16:59:44.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[28259]: audit 2023-07-29T16:59:43.662896+0000 mon.a (mon.0) 207 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7a0e08d-36b3-407d-9d06-6a916d064a34"}]': finished 2023-07-29T16:59:44.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[28259]: cluster 2023-07-29T16:59:43.663098+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-07-29T16:59:44.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:44 smithi078 bash[28259]: audit 2023-07-29T16:59:43.663249+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:59:45.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:45 smithi129 bash[23526]: audit 2023-07-29T16:59:44.283639+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.78:0/3764467081' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:59:45.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:45 smithi078 bash[23704]: audit 2023-07-29T16:59:44.283639+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.78:0/3764467081' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:59:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:45 smithi078 bash[28259]: audit 2023-07-29T16:59:44.283639+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.78:0/3764467081' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:59:46.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:46 smithi129 bash[23526]: cluster 2023-07-29T16:59:45.047218+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:46.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:46 smithi078 bash[23704]: cluster 2023-07-29T16:59:45.047218+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:46.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:46 smithi078 bash[28259]: cluster 2023-07-29T16:59:45.047218+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:48.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:48 smithi129 bash[23526]: cluster 2023-07-29T16:59:47.047743+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:48.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:48 smithi078 bash[23704]: cluster 2023-07-29T16:59:47.047743+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:48.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:48 smithi078 bash[28259]: cluster 2023-07-29T16:59:47.047743+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:50.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:50 smithi129 bash[23526]: cluster 2023-07-29T16:59:49.048182+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:50.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:50 smithi078 bash[23704]: cluster 2023-07-29T16:59:49.048182+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:50.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:50 smithi078 bash[28259]: cluster 2023-07-29T16:59:49.048182+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:52.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[23704]: cluster 2023-07-29T16:59:51.048619+0000 mgr.y (mgr.14144) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:52.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[23704]: audit 2023-07-29T16:59:51.140764+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T16:59:52.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[23704]: audit 2023-07-29T16:59:51.142339+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:52.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[23704]: cephadm 2023-07-29T16:59:51.143533+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Deploying daemon osd.0 on smithi078 2023-07-29T16:59:52.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[28259]: cluster 2023-07-29T16:59:51.048619+0000 mgr.y (mgr.14144) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:52.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[28259]: audit 2023-07-29T16:59:51.140764+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T16:59:52.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[28259]: audit 2023-07-29T16:59:51.142339+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:52.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:52 smithi078 bash[28259]: cephadm 2023-07-29T16:59:51.143533+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Deploying daemon osd.0 on smithi078 2023-07-29T16:59:52.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:52 smithi129 bash[23526]: cluster 2023-07-29T16:59:51.048619+0000 mgr.y (mgr.14144) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:52.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:52 smithi129 bash[23526]: audit 2023-07-29T16:59:51.140764+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T16:59:52.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:52 smithi129 bash[23526]: audit 2023-07-29T16:59:51.142339+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:59:52.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:52 smithi129 bash[23526]: cephadm 2023-07-29T16:59:51.143533+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Deploying daemon osd.0 on smithi078 2023-07-29T16:59:53.526 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2023-07-29T16:59:54.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:54 smithi129 bash[23526]: cluster 2023-07-29T16:59:53.049079+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:54.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:54 smithi129 bash[23526]: audit 2023-07-29T16:59:53.504166+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:54.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:54 smithi129 bash[23526]: audit 2023-07-29T16:59:53.508596+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:54.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:54 smithi129 bash[23526]: audit 2023-07-29T16:59:53.515020+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:54.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[23704]: cluster 2023-07-29T16:59:53.049079+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:54.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[23704]: audit 2023-07-29T16:59:53.504166+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:54.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[23704]: audit 2023-07-29T16:59:53.508596+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:54.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[23704]: audit 2023-07-29T16:59:53.515020+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:54.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[28259]: cluster 2023-07-29T16:59:53.049079+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:54.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[28259]: audit 2023-07-29T16:59:53.504166+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:59:54.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[28259]: audit 2023-07-29T16:59:53.508596+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:54.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:54 smithi078 bash[28259]: audit 2023-07-29T16:59:53.515020+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:54.770 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.0.service 2023-07-29T16:59:54.772 INFO:tasks.cephadm:Deploying osd.1 on smithi078 with /dev/vg_nvme/lv_3... 2023-07-29T16:59:54.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-29T16:59:55.301 INFO:journalctl@ceph.osd.0.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T16:59:55.977 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-29T16:59:55.977 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2023-07-29T16:59:55.977 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2023-07-29T16:59:55.977 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2023-07-29T16:59:55.977 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-29T16:59:55.978 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-29T16:59:55.978 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:55 smithi078 bash[30708]: --> ceph-volume lvm activate successful for osd ID: 0 2023-07-29T16:59:56.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:56 smithi129 bash[23526]: cluster 2023-07-29T16:59:55.049555+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:56.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:56 smithi078 bash[23704]: cluster 2023-07-29T16:59:55.049555+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:56.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:56 smithi078 bash[28259]: cluster 2023-07-29T16:59:55.049555+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:58 smithi078 bash[23704]: cluster 2023-07-29T16:59:57.050069+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 16:59:58 smithi078 bash[23704]: audit 2023-07-29T16:59:57.778371+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:58.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:58 smithi078 bash[28259]: cluster 2023-07-29T16:59:57.050069+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:58.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 16:59:58 smithi078 bash[28259]: audit 2023-07-29T16:59:57.778371+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:58.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:58 smithi129 bash[23526]: cluster 2023-07-29T16:59:57.050069+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:59:58.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 16:59:58 smithi129 bash[23526]: audit 2023-07-29T16:59:57.778371+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T16:59:58.929 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:58 smithi078 bash[30909]: debug 2023-07-29T16:59:58.742+0000 7f5a618ac700 1 -- 172.21.15.78:0/3206384723 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e5774a24e0 con 0x55e57751ac00 2023-07-29T16:59:59.618 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 16:59:59 smithi078 bash[30909]: debug 2023-07-29T16:59:59.286+0000 7f5a69337080 -1 Falling back to public interface 2023-07-29T17:00:00.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:00 smithi078 bash[23704]: cluster 2023-07-29T16:59:59.050628+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:00.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:00 smithi078 bash[23704]: cluster 2023-07-29T17:00:00.000153+0000 mon.a (mon.0) 217 : cluster [INF] overall HEALTH_OK 2023-07-29T17:00:00.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:00 smithi078 bash[28259]: cluster 2023-07-29T16:59:59.050628+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:00.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:00 smithi078 bash[28259]: cluster 2023-07-29T17:00:00.000153+0000 mon.a (mon.0) 217 : cluster [INF] overall HEALTH_OK 2023-07-29T17:00:00.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:00 smithi129 bash[23526]: cluster 2023-07-29T16:59:59.050628+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:00.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:00 smithi129 bash[23526]: cluster 2023-07-29T17:00:00.000153+0000 mon.a (mon.0) 217 : cluster [INF] overall HEALTH_OK 2023-07-29T17:00:00.807 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-29T17:00:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-29T17:00:00.808 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-29T17:00:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-29T17:00:00.808 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0192976 s, 543 MB/s 2023-07-29T17:00:00.808 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-29T17:00:01.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2023-07-29T17:00:02.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:02 smithi078 bash[28259]: cluster 2023-07-29T17:00:01.051166+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:02.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:02 smithi078 bash[23704]: cluster 2023-07-29T17:00:01.051166+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:02.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:02 smithi129 bash[23526]: cluster 2023-07-29T17:00:01.051166+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:02.679 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:00:02 smithi078 bash[30909]: debug 2023-07-29T17:00:02.306+0000 7f5a69337080 -1 osd.0 0 log_to_monitors {default=true} 2023-07-29T17:00:03.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:03 smithi078 bash[23704]: audit 2023-07-29T17:00:02.314454+0000 mon.a (mon.0) 218 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:00:03.384 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:03 smithi078 bash[28259]: audit 2023-07-29T17:00:02.314454+0000 mon.a (mon.0) 218 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:00:03.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:03 smithi129 bash[23526]: audit 2023-07-29T17:00:02.314454+0000 mon.a (mon.0) 218 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:00:04.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:04 smithi129 bash[23526]: cluster 2023-07-29T17:00:03.051672+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:04.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:04 smithi129 bash[23526]: audit 2023-07-29T17:00:03.071793+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T17:00:04.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:04 smithi129 bash[23526]: cluster 2023-07-29T17:00:03.071999+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-07-29T17:00:04.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:04 smithi129 bash[23526]: audit 2023-07-29T17:00:03.072229+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:04.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:04 smithi129 bash[23526]: audit 2023-07-29T17:00:03.072697+0000 mon.a (mon.0) 222 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:04.429 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[30909]: debug 2023-07-29T17:00:04.082+0000 7f5a5f8a8700 -1 osd.0 0 waiting for initial osdmap 2023-07-29T17:00:04.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[28259]: cluster 2023-07-29T17:00:03.051672+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:04.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[28259]: audit 2023-07-29T17:00:03.071793+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T17:00:04.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[28259]: cluster 2023-07-29T17:00:03.071999+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-07-29T17:00:04.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[28259]: audit 2023-07-29T17:00:03.072229+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:04.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[28259]: audit 2023-07-29T17:00:03.072697+0000 mon.a (mon.0) 222 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:04.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[23704]: cluster 2023-07-29T17:00:03.051672+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:04.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[23704]: audit 2023-07-29T17:00:03.071793+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T17:00:04.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[23704]: cluster 2023-07-29T17:00:03.071999+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-07-29T17:00:04.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[23704]: audit 2023-07-29T17:00:03.072229+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:04.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:04 smithi078 bash[23704]: audit 2023-07-29T17:00:03.072697+0000 mon.a (mon.0) 222 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:05.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:05 smithi129 bash[23526]: audit 2023-07-29T17:00:04.075746+0000 mon.a (mon.0) 223 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:05.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:05 smithi129 bash[23526]: cluster 2023-07-29T17:00:04.075965+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-07-29T17:00:05.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:05 smithi129 bash[23526]: audit 2023-07-29T17:00:04.076857+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:05.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:05 smithi129 bash[23526]: audit 2023-07-29T17:00:04.085266+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:05.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:05 smithi129 bash[23526]: audit 2023-07-29T17:00:05.032535+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:05.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[23704]: audit 2023-07-29T17:00:04.075746+0000 mon.a (mon.0) 223 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:05.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[23704]: cluster 2023-07-29T17:00:04.075965+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-07-29T17:00:05.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[23704]: audit 2023-07-29T17:00:04.076857+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:05.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[23704]: audit 2023-07-29T17:00:04.085266+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:05.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[23704]: audit 2023-07-29T17:00:05.032535+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:05.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[28259]: audit 2023-07-29T17:00:04.075746+0000 mon.a (mon.0) 223 : audit [INF] from='osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:05.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[28259]: cluster 2023-07-29T17:00:04.075965+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-07-29T17:00:05.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[28259]: audit 2023-07-29T17:00:04.076857+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:05.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[28259]: audit 2023-07-29T17:00:04.085266+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:05.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:05 smithi078 bash[28259]: audit 2023-07-29T17:00:05.032535+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: cluster 2023-07-29T17:00:03.352093+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:06.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: cluster 2023-07-29T17:00:03.352298+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:06.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: cephadm 2023-07-29T17:00:05.025960+0000 mgr.y (mgr.14144) 61 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: cluster 2023-07-29T17:00:05.052057+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: cluster 2023-07-29T17:00:05.086681+0000 mon.a (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458] boot 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: cluster 2023-07-29T17:00:05.086758+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.087681+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.290077+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.292912+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.569962+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.589867+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:06.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.592808+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:06.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[23704]: audit 2023-07-29T17:00:05.594307+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:06.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: cluster 2023-07-29T17:00:03.352093+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:06.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: cluster 2023-07-29T17:00:03.352298+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: cephadm 2023-07-29T17:00:05.025960+0000 mgr.y (mgr.14144) 61 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: cluster 2023-07-29T17:00:05.052057+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: cluster 2023-07-29T17:00:05.086681+0000 mon.a (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458] boot 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: cluster 2023-07-29T17:00:05.086758+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.087681+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.290077+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.292912+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:06.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.569962+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.589867+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:06.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.592808+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:06.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:06 smithi129 bash[23526]: audit 2023-07-29T17:00:05.594307+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:06.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: cluster 2023-07-29T17:00:03.352093+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:06.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: cluster 2023-07-29T17:00:03.352298+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:06.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: cephadm 2023-07-29T17:00:05.025960+0000 mgr.y (mgr.14144) 61 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: cluster 2023-07-29T17:00:05.052057+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: cluster 2023-07-29T17:00:05.086681+0000 mon.a (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.78:6802/3465957458,v1:172.21.15.78:6803/3465957458] boot 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: cluster 2023-07-29T17:00:05.086758+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.087681+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.290077+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.292912+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:06.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.569962+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:06.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.589867+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:06.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.592808+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:06.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:06 smithi078 bash[28259]: audit 2023-07-29T17:00:05.594307+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:07.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:07 smithi129 bash[23526]: audit 2023-07-29T17:00:05.588837+0000 mgr.y (mgr.14144) 63 : audit [DBG] from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:07.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:07 smithi129 bash[23526]: cephadm 2023-07-29T17:00:05.592382+0000 mgr.y (mgr.14144) 64 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:07.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:07 smithi129 bash[23526]: cluster 2023-07-29T17:00:06.108125+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-07-29T17:00:07.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:07 smithi078 bash[23704]: audit 2023-07-29T17:00:05.588837+0000 mgr.y (mgr.14144) 63 : audit [DBG] from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:07.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:07 smithi078 bash[23704]: cephadm 2023-07-29T17:00:05.592382+0000 mgr.y (mgr.14144) 64 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:07.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:07 smithi078 bash[23704]: cluster 2023-07-29T17:00:06.108125+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-07-29T17:00:07.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:07 smithi078 bash[28259]: audit 2023-07-29T17:00:05.588837+0000 mgr.y (mgr.14144) 63 : audit [DBG] from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:07.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:07 smithi078 bash[28259]: cephadm 2023-07-29T17:00:05.592382+0000 mgr.y (mgr.14144) 64 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:07.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:07 smithi078 bash[28259]: cluster 2023-07-29T17:00:06.108125+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-07-29T17:00:08.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:08 smithi129 bash[23526]: cluster 2023-07-29T17:00:07.052633+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:08.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:08 smithi078 bash[23704]: cluster 2023-07-29T17:00:07.052633+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:08.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:08 smithi078 bash[28259]: cluster 2023-07-29T17:00:07.052633+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:10.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:10 smithi129 bash[23526]: cluster 2023-07-29T17:00:09.053245+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:10.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:10 smithi129 bash[23526]: audit 2023-07-29T17:00:09.396580+0000 mon.a (mon.0) 238 : audit [INF] from='client.? 172.21.15.78:0/3765331251' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "567bfd6c-b9b4-4b44-994a-fa00e3cd0703"}]: dispatch 2023-07-29T17:00:10.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:10 smithi129 bash[23526]: audit 2023-07-29T17:00:09.402097+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.78:0/3765331251' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "567bfd6c-b9b4-4b44-994a-fa00e3cd0703"}]': finished 2023-07-29T17:00:10.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:10 smithi129 bash[23526]: cluster 2023-07-29T17:00:09.402180+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-07-29T17:00:10.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:10 smithi129 bash[23526]: audit 2023-07-29T17:00:09.402423+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:10.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:10 smithi129 bash[23526]: audit 2023-07-29T17:00:10.021590+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? 172.21.15.78:0/3491952299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:00:10.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[28259]: cluster 2023-07-29T17:00:09.053245+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:10.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[28259]: audit 2023-07-29T17:00:09.396580+0000 mon.a (mon.0) 238 : audit [INF] from='client.? 172.21.15.78:0/3765331251' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "567bfd6c-b9b4-4b44-994a-fa00e3cd0703"}]: dispatch 2023-07-29T17:00:10.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[28259]: audit 2023-07-29T17:00:09.402097+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.78:0/3765331251' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "567bfd6c-b9b4-4b44-994a-fa00e3cd0703"}]': finished 2023-07-29T17:00:10.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[28259]: cluster 2023-07-29T17:00:09.402180+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-07-29T17:00:10.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[28259]: audit 2023-07-29T17:00:09.402423+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:10.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[28259]: audit 2023-07-29T17:00:10.021590+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? 172.21.15.78:0/3491952299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:00:10.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[23704]: cluster 2023-07-29T17:00:09.053245+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:10.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[23704]: audit 2023-07-29T17:00:09.396580+0000 mon.a (mon.0) 238 : audit [INF] from='client.? 172.21.15.78:0/3765331251' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "567bfd6c-b9b4-4b44-994a-fa00e3cd0703"}]: dispatch 2023-07-29T17:00:10.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[23704]: audit 2023-07-29T17:00:09.402097+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.78:0/3765331251' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "567bfd6c-b9b4-4b44-994a-fa00e3cd0703"}]': finished 2023-07-29T17:00:10.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[23704]: cluster 2023-07-29T17:00:09.402180+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-07-29T17:00:10.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[23704]: audit 2023-07-29T17:00:09.402423+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:10.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:10 smithi078 bash[23704]: audit 2023-07-29T17:00:10.021590+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? 172.21.15.78:0/3491952299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:00:12.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:12 smithi129 bash[23526]: cluster 2023-07-29T17:00:11.053722+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:12.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:12 smithi078 bash[23704]: cluster 2023-07-29T17:00:11.053722+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:12.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:12 smithi078 bash[28259]: cluster 2023-07-29T17:00:11.053722+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:14.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:14 smithi129 bash[23526]: cluster 2023-07-29T17:00:13.054266+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:14.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:14 smithi078 bash[23704]: cluster 2023-07-29T17:00:13.054266+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:14.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:14 smithi078 bash[28259]: cluster 2023-07-29T17:00:13.054266+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:16.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:16 smithi129 bash[23526]: cluster 2023-07-29T17:00:15.054722+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:16.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:16 smithi078 bash[23704]: cluster 2023-07-29T17:00:15.054722+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:16.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:16 smithi078 bash[28259]: cluster 2023-07-29T17:00:15.054722+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:17.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:17 smithi129 bash[23526]: audit 2023-07-29T17:00:17.060253+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:00:17.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:17 smithi129 bash[23526]: audit 2023-07-29T17:00:17.061868+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:17.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:17 smithi078 bash[28259]: audit 2023-07-29T17:00:17.060253+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:00:17.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:17 smithi078 bash[28259]: audit 2023-07-29T17:00:17.061868+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:17.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:17 smithi078 bash[23704]: audit 2023-07-29T17:00:17.060253+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:00:17.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:17 smithi078 bash[23704]: audit 2023-07-29T17:00:17.061868+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:18.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:18 smithi129 bash[23526]: cluster 2023-07-29T17:00:17.055140+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:18.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:18 smithi129 bash[23526]: cephadm 2023-07-29T17:00:17.063117+0000 mgr.y (mgr.14144) 71 : cephadm [INF] Deploying daemon osd.1 on smithi078 2023-07-29T17:00:18.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:18 smithi078 bash[23704]: cluster 2023-07-29T17:00:17.055140+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:18.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:18 smithi078 bash[23704]: cephadm 2023-07-29T17:00:17.063117+0000 mgr.y (mgr.14144) 71 : cephadm [INF] Deploying daemon osd.1 on smithi078 2023-07-29T17:00:18.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:18 smithi078 bash[28259]: cluster 2023-07-29T17:00:17.055140+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:18.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:18 smithi078 bash[28259]: cephadm 2023-07-29T17:00:17.063117+0000 mgr.y (mgr.14144) 71 : cephadm [INF] Deploying daemon osd.1 on smithi078 2023-07-29T17:00:19.507 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 1 on host 'smithi078' 2023-07-29T17:00:20.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:20 smithi129 bash[23526]: cluster 2023-07-29T17:00:19.055590+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:20.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:20 smithi129 bash[23526]: audit 2023-07-29T17:00:19.486252+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:20.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:20 smithi129 bash[23526]: audit 2023-07-29T17:00:19.491253+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:20.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:20 smithi129 bash[23526]: audit 2023-07-29T17:00:19.499755+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:20.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[23704]: cluster 2023-07-29T17:00:19.055590+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:20.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[23704]: audit 2023-07-29T17:00:19.486252+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:20.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[23704]: audit 2023-07-29T17:00:19.491253+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:20.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[23704]: audit 2023-07-29T17:00:19.499755+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:20.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[28259]: cluster 2023-07-29T17:00:19.055590+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:20.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[28259]: audit 2023-07-29T17:00:19.486252+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:20.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[28259]: audit 2023-07-29T17:00:19.491253+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:20.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:20 smithi078 bash[28259]: audit 2023-07-29T17:00:19.499755+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:20.753 DEBUG:teuthology.orchestra.run.smithi078:osd.1> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.1.service 2023-07-29T17:00:20.756 INFO:tasks.cephadm:Deploying osd.2 on smithi078 with /dev/vg_nvme/lv_2... 2023-07-29T17:00:20.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-29T17:00:21.310 INFO:journalctl@ceph.osd.1.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:00:22.163 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-29T17:00:22.163 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2023-07-29T17:00:22.163 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2023-07-29T17:00:22.163 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2023-07-29T17:00:22.164 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-29T17:00:22.164 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-29T17:00:22.164 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:21 smithi078 bash[33463]: --> ceph-volume lvm activate successful for osd ID: 1 2023-07-29T17:00:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:22 smithi078 bash[23704]: cluster 2023-07-29T17:00:21.056033+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:22.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:22 smithi078 bash[28259]: cluster 2023-07-29T17:00:21.056033+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:22.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:22 smithi129 bash[23526]: cluster 2023-07-29T17:00:21.056033+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:24.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:24 smithi078 bash[28259]: cluster 2023-07-29T17:00:23.056563+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:24.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:24 smithi078 bash[23704]: cluster 2023-07-29T17:00:23.056563+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:24.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:24 smithi129 bash[23526]: cluster 2023-07-29T17:00:23.056563+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:24.680 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:24 smithi078 bash[33691]: debug 2023-07-29T17:00:24.523+0000 7ff72ae37700 1 -- 172.21.15.78:0/3672078668 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5632243f24e0 con 0x56322446b400 2023-07-29T17:00:25.179 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:24 smithi078 bash[33691]: debug 2023-07-29T17:00:24.815+0000 7ff7328c2080 -1 Falling back to public interface 2023-07-29T17:00:25.380 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-29T17:00:25.381 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-29T17:00:25.381 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-29T17:00:25.381 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-29T17:00:25.381 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0176661 s, 594 MB/s 2023-07-29T17:00:25.382 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-29T17:00:26.160 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:25 smithi129 bash[23526]: audit 2023-07-29T17:00:24.685428+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:26.160 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:25 smithi129 bash[23526]: cluster 2023-07-29T17:00:25.056961+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:26.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:25 smithi078 bash[23704]: audit 2023-07-29T17:00:24.685428+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:26.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:25 smithi078 bash[23704]: cluster 2023-07-29T17:00:25.056961+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:26.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:25 smithi078 bash[28259]: audit 2023-07-29T17:00:24.685428+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:26.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:25 smithi078 bash[28259]: cluster 2023-07-29T17:00:25.056961+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:26.377 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2023-07-29T17:00:28.088 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:27 smithi078 bash[33691]: debug 2023-07-29T17:00:27.803+0000 7ff7328c2080 -1 osd.1 0 log_to_monitors {default=true} 2023-07-29T17:00:28.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:28 smithi078 bash[23704]: cluster 2023-07-29T17:00:27.057429+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:28.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:28 smithi078 bash[23704]: audit 2023-07-29T17:00:27.812623+0000 mon.c (mon.2) 6 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:00:28.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:28 smithi078 bash[23704]: audit 2023-07-29T17:00:27.813291+0000 mon.a (mon.0) 248 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:00:28.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:28 smithi078 bash[28259]: cluster 2023-07-29T17:00:27.057429+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:28.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:28 smithi078 bash[28259]: audit 2023-07-29T17:00:27.812623+0000 mon.c (mon.2) 6 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:00:28.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:28 smithi078 bash[28259]: audit 2023-07-29T17:00:27.813291+0000 mon.a (mon.0) 248 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:00:28.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:28 smithi129 bash[23526]: cluster 2023-07-29T17:00:27.057429+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:28.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:28 smithi129 bash[23526]: audit 2023-07-29T17:00:27.812623+0000 mon.c (mon.2) 6 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:00:28.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:28 smithi129 bash[23526]: audit 2023-07-29T17:00:27.813291+0000 mon.a (mon.0) 248 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:00:29.134 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[33691]: debug 2023-07-29T17:00:29.091+0000 7ff728e33700 -1 osd.1 0 waiting for initial osdmap 2023-07-29T17:00:29.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[28259]: audit 2023-07-29T17:00:28.077020+0000 mon.a (mon.0) 249 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T17:00:29.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[28259]: cluster 2023-07-29T17:00:28.077208+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-07-29T17:00:29.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[28259]: audit 2023-07-29T17:00:28.077954+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:29.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[28259]: audit 2023-07-29T17:00:28.080957+0000 mon.c (mon.2) 7 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:29.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[28259]: audit 2023-07-29T17:00:28.086370+0000 mon.a (mon.0) 252 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:29.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[23704]: audit 2023-07-29T17:00:28.077020+0000 mon.a (mon.0) 249 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T17:00:29.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[23704]: cluster 2023-07-29T17:00:28.077208+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-07-29T17:00:29.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[23704]: audit 2023-07-29T17:00:28.077954+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:29.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[23704]: audit 2023-07-29T17:00:28.080957+0000 mon.c (mon.2) 7 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:29.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:29 smithi078 bash[23704]: audit 2023-07-29T17:00:28.086370+0000 mon.a (mon.0) 252 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:29.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:29 smithi129 bash[23526]: audit 2023-07-29T17:00:28.077020+0000 mon.a (mon.0) 249 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T17:00:29.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:29 smithi129 bash[23526]: cluster 2023-07-29T17:00:28.077208+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-07-29T17:00:29.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:29 smithi129 bash[23526]: audit 2023-07-29T17:00:28.077954+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:29.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:29 smithi129 bash[23526]: audit 2023-07-29T17:00:28.080957+0000 mon.c (mon.2) 7 : audit [INF] from='osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:29.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:29 smithi129 bash[23526]: audit 2023-07-29T17:00:28.086370+0000 mon.a (mon.0) 252 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:30.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[23704]: cluster 2023-07-29T17:00:29.057931+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:30.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[23704]: audit 2023-07-29T17:00:29.088449+0000 mon.a (mon.0) 253 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:30.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[23704]: cluster 2023-07-29T17:00:29.088699+0000 mon.a (mon.0) 254 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-07-29T17:00:30.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[23704]: audit 2023-07-29T17:00:29.089566+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:30.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[23704]: audit 2023-07-29T17:00:29.096708+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:30.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[28259]: cluster 2023-07-29T17:00:29.057931+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:30.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[28259]: audit 2023-07-29T17:00:29.088449+0000 mon.a (mon.0) 253 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:30.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[28259]: cluster 2023-07-29T17:00:29.088699+0000 mon.a (mon.0) 254 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-07-29T17:00:30.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[28259]: audit 2023-07-29T17:00:29.089566+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:30.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:30 smithi078 bash[28259]: audit 2023-07-29T17:00:29.096708+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:30.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:30 smithi129 bash[23526]: cluster 2023-07-29T17:00:29.057931+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:30.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:30 smithi129 bash[23526]: audit 2023-07-29T17:00:29.088449+0000 mon.a (mon.0) 253 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:30.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:30 smithi129 bash[23526]: cluster 2023-07-29T17:00:29.088699+0000 mon.a (mon.0) 254 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-07-29T17:00:30.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:30 smithi129 bash[23526]: audit 2023-07-29T17:00:29.089566+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:30.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:30 smithi129 bash[23526]: audit 2023-07-29T17:00:29.096708+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:31.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: cluster 2023-07-29T17:00:28.798301+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:31.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: cluster 2023-07-29T17:00:28.798517+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:31.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: cluster 2023-07-29T17:00:30.099480+0000 mon.a (mon.0) 257 : cluster [INF] osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047] boot 2023-07-29T17:00:31.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: cluster 2023-07-29T17:00:30.099629+0000 mon.a (mon.0) 258 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-07-29T17:00:31.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: audit 2023-07-29T17:00:30.100543+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:31.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: audit 2023-07-29T17:00:30.771931+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:31.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: audit 2023-07-29T17:00:30.775096+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:31.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:31 smithi129 bash[23526]: audit 2023-07-29T17:00:30.776638+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:31.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: cluster 2023-07-29T17:00:28.798301+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:31.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: cluster 2023-07-29T17:00:28.798517+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:31.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: cluster 2023-07-29T17:00:30.099480+0000 mon.a (mon.0) 257 : cluster [INF] osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047] boot 2023-07-29T17:00:31.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: cluster 2023-07-29T17:00:30.099629+0000 mon.a (mon.0) 258 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-07-29T17:00:31.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: audit 2023-07-29T17:00:30.100543+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: audit 2023-07-29T17:00:30.771931+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: audit 2023-07-29T17:00:30.775096+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[23704]: audit 2023-07-29T17:00:30.776638+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: cluster 2023-07-29T17:00:28.798301+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: cluster 2023-07-29T17:00:28.798517+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: cluster 2023-07-29T17:00:30.099480+0000 mon.a (mon.0) 257 : cluster [INF] osd.1 [v2:172.21.15.78:6810/3824985047,v1:172.21.15.78:6811/3824985047] boot 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: cluster 2023-07-29T17:00:30.099629+0000 mon.a (mon.0) 258 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: audit 2023-07-29T17:00:30.100543+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: audit 2023-07-29T17:00:30.771931+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: audit 2023-07-29T17:00:30.775096+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:31.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:31 smithi078 bash[28259]: audit 2023-07-29T17:00:30.776638+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:32.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: audit 2023-07-29T17:00:30.770874+0000 mgr.y (mgr.14144) 78 : audit [DBG] from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:32.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: cephadm 2023-07-29T17:00:30.774482+0000 mgr.y (mgr.14144) 79 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:32.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: cluster 2023-07-29T17:00:31.058399+0000 mgr.y (mgr.14144) 80 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:32.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: cluster 2023-07-29T17:00:31.116636+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-07-29T17:00:32.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: audit 2023-07-29T17:00:31.996579+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: audit 2023-07-29T17:00:32.004088+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: audit 2023-07-29T17:00:32.006355+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:32.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:32 smithi129 bash[23526]: audit 2023-07-29T17:00:32.013314+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: audit 2023-07-29T17:00:30.770874+0000 mgr.y (mgr.14144) 78 : audit [DBG] from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:32.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: cephadm 2023-07-29T17:00:30.774482+0000 mgr.y (mgr.14144) 79 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:32.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: cluster 2023-07-29T17:00:31.058399+0000 mgr.y (mgr.14144) 80 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:32.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: cluster 2023-07-29T17:00:31.116636+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-07-29T17:00:32.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: audit 2023-07-29T17:00:31.996579+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: audit 2023-07-29T17:00:32.004088+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: audit 2023-07-29T17:00:32.006355+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:32.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[23704]: audit 2023-07-29T17:00:32.013314+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: audit 2023-07-29T17:00:30.770874+0000 mgr.y (mgr.14144) 78 : audit [DBG] from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: cephadm 2023-07-29T17:00:30.774482+0000 mgr.y (mgr.14144) 79 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: cluster 2023-07-29T17:00:31.058399+0000 mgr.y (mgr.14144) 80 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: cluster 2023-07-29T17:00:31.116636+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: audit 2023-07-29T17:00:31.996579+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: audit 2023-07-29T17:00:32.004088+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: audit 2023-07-29T17:00:32.006355+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:32.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:32 smithi078 bash[28259]: audit 2023-07-29T17:00:32.013314+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:33 smithi129 bash[23526]: cephadm 2023-07-29T17:00:31.987988+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:33.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:33 smithi078 bash[23704]: cephadm 2023-07-29T17:00:31.987988+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:33.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:33 smithi078 bash[28259]: cephadm 2023-07-29T17:00:31.987988+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:34.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:34 smithi129 bash[23526]: cluster 2023-07-29T17:00:33.058934+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:34.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:34 smithi078 bash[23704]: cluster 2023-07-29T17:00:33.058934+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:34.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:34 smithi078 bash[28259]: cluster 2023-07-29T17:00:33.058934+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:35.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:35 smithi129 bash[23526]: audit 2023-07-29T17:00:34.643667+0000 mon.a (mon.0) 268 : audit [INF] from='client.? 172.21.15.78:0/1320959134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0714ad7-47d8-4b54-bcb4-80ebe0a96310"}]: dispatch 2023-07-29T17:00:35.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:35 smithi129 bash[23526]: audit 2023-07-29T17:00:34.651552+0000 mon.a (mon.0) 269 : audit [INF] from='client.? 172.21.15.78:0/1320959134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0714ad7-47d8-4b54-bcb4-80ebe0a96310"}]': finished 2023-07-29T17:00:35.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:35 smithi129 bash[23526]: cluster 2023-07-29T17:00:34.651708+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-07-29T17:00:35.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:35 smithi129 bash[23526]: audit 2023-07-29T17:00:34.651837+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:35.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[23704]: audit 2023-07-29T17:00:34.643667+0000 mon.a (mon.0) 268 : audit [INF] from='client.? 172.21.15.78:0/1320959134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0714ad7-47d8-4b54-bcb4-80ebe0a96310"}]: dispatch 2023-07-29T17:00:35.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[23704]: audit 2023-07-29T17:00:34.651552+0000 mon.a (mon.0) 269 : audit [INF] from='client.? 172.21.15.78:0/1320959134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0714ad7-47d8-4b54-bcb4-80ebe0a96310"}]': finished 2023-07-29T17:00:35.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[23704]: cluster 2023-07-29T17:00:34.651708+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-07-29T17:00:35.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[23704]: audit 2023-07-29T17:00:34.651837+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:35.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[28259]: audit 2023-07-29T17:00:34.643667+0000 mon.a (mon.0) 268 : audit [INF] from='client.? 172.21.15.78:0/1320959134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0714ad7-47d8-4b54-bcb4-80ebe0a96310"}]: dispatch 2023-07-29T17:00:35.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[28259]: audit 2023-07-29T17:00:34.651552+0000 mon.a (mon.0) 269 : audit [INF] from='client.? 172.21.15.78:0/1320959134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0714ad7-47d8-4b54-bcb4-80ebe0a96310"}]': finished 2023-07-29T17:00:35.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[28259]: cluster 2023-07-29T17:00:34.651708+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-07-29T17:00:35.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:35 smithi078 bash[28259]: audit 2023-07-29T17:00:34.651837+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:36.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:36 smithi129 bash[23526]: cluster 2023-07-29T17:00:35.059429+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:36 smithi129 bash[23526]: audit 2023-07-29T17:00:35.241082+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:00:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:36 smithi129 bash[23526]: audit 2023-07-29T17:00:35.243193+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:00:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:36 smithi129 bash[23526]: audit 2023-07-29T17:00:35.342878+0000 mon.a (mon.0) 274 : audit [DBG] from='client.? 172.21.15.78:0/2411471475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:00:36.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[23704]: cluster 2023-07-29T17:00:35.059429+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:36.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[23704]: audit 2023-07-29T17:00:35.241082+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:00:36.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[23704]: audit 2023-07-29T17:00:35.243193+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:00:36.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[23704]: audit 2023-07-29T17:00:35.342878+0000 mon.a (mon.0) 274 : audit [DBG] from='client.? 172.21.15.78:0/2411471475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:00:36.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[28259]: cluster 2023-07-29T17:00:35.059429+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:36.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[28259]: audit 2023-07-29T17:00:35.241082+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:00:36.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[28259]: audit 2023-07-29T17:00:35.243193+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:00:36.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:36 smithi078 bash[28259]: audit 2023-07-29T17:00:35.342878+0000 mon.a (mon.0) 274 : audit [DBG] from='client.? 172.21.15.78:0/2411471475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:00:38.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:38 smithi129 bash[23526]: cluster 2023-07-29T17:00:37.059852+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:38.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:38 smithi078 bash[23704]: cluster 2023-07-29T17:00:37.059852+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:38.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:38 smithi078 bash[28259]: cluster 2023-07-29T17:00:37.059852+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:40.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:40 smithi129 bash[23526]: cluster 2023-07-29T17:00:39.060344+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:40.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:40 smithi078 bash[23704]: cluster 2023-07-29T17:00:39.060344+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:40.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:40 smithi078 bash[28259]: cluster 2023-07-29T17:00:39.060344+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:42.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:42 smithi129 bash[23526]: cluster 2023-07-29T17:00:41.060901+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:42.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:42 smithi078 bash[23704]: cluster 2023-07-29T17:00:41.060901+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:42.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:42 smithi078 bash[28259]: cluster 2023-07-29T17:00:41.060901+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:43.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:43 smithi078 bash[23704]: audit 2023-07-29T17:00:42.454473+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:00:43.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:43 smithi078 bash[23704]: audit 2023-07-29T17:00:42.456283+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:43.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:43 smithi078 bash[28259]: audit 2023-07-29T17:00:42.454473+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:00:43.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:43 smithi078 bash[28259]: audit 2023-07-29T17:00:42.456283+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:43.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:43 smithi129 bash[23526]: audit 2023-07-29T17:00:42.454473+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:00:43.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:43 smithi129 bash[23526]: audit 2023-07-29T17:00:42.456283+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:44.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:44 smithi078 bash[23704]: cephadm 2023-07-29T17:00:42.457802+0000 mgr.y (mgr.14144) 87 : cephadm [INF] Deploying daemon osd.2 on smithi078 2023-07-29T17:00:44.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:44 smithi078 bash[23704]: cluster 2023-07-29T17:00:43.061439+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:44.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:44 smithi078 bash[28259]: cephadm 2023-07-29T17:00:42.457802+0000 mgr.y (mgr.14144) 87 : cephadm [INF] Deploying daemon osd.2 on smithi078 2023-07-29T17:00:44.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:44 smithi078 bash[28259]: cluster 2023-07-29T17:00:43.061439+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:44.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:44 smithi129 bash[23526]: cephadm 2023-07-29T17:00:42.457802+0000 mgr.y (mgr.14144) 87 : cephadm [INF] Deploying daemon osd.2 on smithi078 2023-07-29T17:00:44.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:44 smithi129 bash[23526]: cluster 2023-07-29T17:00:43.061439+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:44.925 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 2 on host 'smithi078' 2023-07-29T17:00:45.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:45 smithi078 bash[23704]: audit 2023-07-29T17:00:44.905200+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:45.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:45 smithi078 bash[23704]: audit 2023-07-29T17:00:44.910968+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:45.369 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:45 smithi078 bash[23704]: audit 2023-07-29T17:00:44.920343+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:45.369 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:45 smithi078 bash[28259]: audit 2023-07-29T17:00:44.905200+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:45.369 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:45 smithi078 bash[28259]: audit 2023-07-29T17:00:44.910968+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:45.370 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:45 smithi078 bash[28259]: audit 2023-07-29T17:00:44.920343+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:45.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:45 smithi129 bash[23526]: audit 2023-07-29T17:00:44.905200+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:45.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:45 smithi129 bash[23526]: audit 2023-07-29T17:00:44.910968+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:45.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:45 smithi129 bash[23526]: audit 2023-07-29T17:00:44.920343+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:46.274 DEBUG:teuthology.orchestra.run.smithi078:osd.2> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.2.service 2023-07-29T17:00:46.277 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_1... 2023-07-29T17:00:46.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-29T17:00:46.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:46 smithi078 bash[23704]: cluster 2023-07-29T17:00:45.061882+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:46.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:46 smithi078 bash[28259]: cluster 2023-07-29T17:00:45.061882+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:46.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:46 smithi129 bash[23526]: cluster 2023-07-29T17:00:45.061882+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:46.815 INFO:journalctl@ceph.osd.2.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:00:47.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[28259]: cluster 2023-07-29T17:00:47.062334+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:47.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[23704]: cluster 2023-07-29T17:00:47.062334+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:47.504 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-29T17:00:47.504 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2023-07-29T17:00:47.504 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2023-07-29T17:00:47.504 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2023-07-29T17:00:47.504 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-29T17:00:47.504 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-29T17:00:47.505 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:47 smithi078 bash[36223]: --> ceph-volume lvm activate successful for osd ID: 2 2023-07-29T17:00:47.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:47 smithi129 bash[23526]: cluster 2023-07-29T17:00:47.062334+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:50.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:50 smithi129 bash[23526]: cluster 2023-07-29T17:00:49.063018+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:50.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:50 smithi129 bash[23526]: audit 2023-07-29T17:00:49.259946+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:50.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:50 smithi078 bash[23704]: cluster 2023-07-29T17:00:49.063018+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:50.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:50 smithi078 bash[23704]: audit 2023-07-29T17:00:49.259946+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:50.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:50 smithi078 bash[28259]: cluster 2023-07-29T17:00:49.063018+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:50.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:50 smithi078 bash[28259]: audit 2023-07-29T17:00:49.259946+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:50.926 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:50 smithi078 bash[36468]: debug 2023-07-29T17:00:50.616+0000 7fc764591700 1 -- 172.21.15.78:0/4013905548 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5572481084e0 con 0x557248180c00 2023-07-29T17:00:51.461 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:51 smithi078 bash[36468]: debug 2023-07-29T17:00:51.172+0000 7fc76c01c080 -1 Falling back to public interface 2023-07-29T17:00:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-29T17:00:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-29T17:00:52.115 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-29T17:00:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-29T17:00:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0183539 s, 571 MB/s 2023-07-29T17:00:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-29T17:00:52.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:52 smithi129 bash[23526]: cluster 2023-07-29T17:00:51.063599+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:52.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:52 smithi078 bash[23704]: cluster 2023-07-29T17:00:51.063599+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:52.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:52 smithi078 bash[28259]: cluster 2023-07-29T17:00:51.063599+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:53.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2023-07-29T17:00:54.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:54 smithi129 bash[23526]: cluster 2023-07-29T17:00:53.064093+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:54.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:54 smithi078 bash[23704]: cluster 2023-07-29T17:00:53.064093+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:54.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:54 smithi078 bash[28259]: cluster 2023-07-29T17:00:53.064093+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:54.426 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:54 smithi078 bash[36468]: debug 2023-07-29T17:00:54.156+0000 7fc76c01c080 -1 osd.2 0 log_to_monitors {default=true} 2023-07-29T17:00:55.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:55 smithi078 bash[28259]: audit 2023-07-29T17:00:54.166251+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:00:55.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:55 smithi078 bash[28259]: audit 2023-07-29T17:00:54.166775+0000 mon.a (mon.0) 281 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:00:55.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:55 smithi078 bash[23704]: audit 2023-07-29T17:00:54.166251+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:00:55.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:55 smithi078 bash[23704]: audit 2023-07-29T17:00:54.166775+0000 mon.a (mon.0) 281 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:00:55.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:55 smithi129 bash[23526]: audit 2023-07-29T17:00:54.166251+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:00:55.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:55 smithi129 bash[23526]: audit 2023-07-29T17:00:54.166775+0000 mon.a (mon.0) 281 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:00:56.426 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[36468]: debug 2023-07-29T17:00:56.156+0000 7fc76258d700 -1 osd.2 0 waiting for initial osdmap 2023-07-29T17:00:56.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[23704]: cluster 2023-07-29T17:00:55.064658+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:56.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[23704]: audit 2023-07-29T17:00:55.145021+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T17:00:56.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[23704]: cluster 2023-07-29T17:00:55.145224+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-07-29T17:00:56.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[23704]: audit 2023-07-29T17:00:55.145416+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:56.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[23704]: audit 2023-07-29T17:00:55.146468+0000 mon.c (mon.2) 9 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:56.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[23704]: audit 2023-07-29T17:00:55.147060+0000 mon.a (mon.0) 285 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:56.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[28259]: cluster 2023-07-29T17:00:55.064658+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:56.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[28259]: audit 2023-07-29T17:00:55.145021+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T17:00:56.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[28259]: cluster 2023-07-29T17:00:55.145224+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-07-29T17:00:56.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[28259]: audit 2023-07-29T17:00:55.145416+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:56.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[28259]: audit 2023-07-29T17:00:55.146468+0000 mon.c (mon.2) 9 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:56.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:56 smithi078 bash[28259]: audit 2023-07-29T17:00:55.147060+0000 mon.a (mon.0) 285 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:56.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:56 smithi129 bash[23526]: cluster 2023-07-29T17:00:55.064658+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:56.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:56 smithi129 bash[23526]: audit 2023-07-29T17:00:55.145021+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T17:00:56.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:56 smithi129 bash[23526]: cluster 2023-07-29T17:00:55.145224+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-07-29T17:00:56.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:56 smithi129 bash[23526]: audit 2023-07-29T17:00:55.145416+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:56.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:56 smithi129 bash[23526]: audit 2023-07-29T17:00:55.146468+0000 mon.c (mon.2) 9 : audit [INF] from='osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:56.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:56 smithi129 bash[23526]: audit 2023-07-29T17:00:55.147060+0000 mon.a (mon.0) 285 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:00:57.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.150494+0000 mon.a (mon.0) 286 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:57.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: cluster 2023-07-29T17:00:56.150717+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-07-29T17:00:57.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.151544+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:57.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.158992+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:57.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.746308+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.752982+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.755374+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:57.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:56.760783+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:57.134550+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:57.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:57.137558+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:57.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[23704]: audit 2023-07-29T17:00:57.138857+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.150494+0000 mon.a (mon.0) 286 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: cluster 2023-07-29T17:00:56.150717+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.151544+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.158992+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.746308+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.752982+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.755374+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:57.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:56.760783+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:57.134550+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:57.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:57.137558+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:57.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:57 smithi078 bash[28259]: audit 2023-07-29T17:00:57.138857+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:57.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.150494+0000 mon.a (mon.0) 286 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:00:57.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: cluster 2023-07-29T17:00:56.150717+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-07-29T17:00:57.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.151544+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:57.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.158992+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:57.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.746308+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.752982+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.755374+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:00:57.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:56.760783+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:00:57.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:57.134550+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:00:57.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:57.137558+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:00:57.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:57 smithi129 bash[23526]: audit 2023-07-29T17:00:57.138857+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:00:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cluster 2023-07-29T17:00:55.136938+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:58.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cluster 2023-07-29T17:00:55.137201+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:58.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cephadm 2023-07-29T17:00:56.740906+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:58.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cluster 2023-07-29T17:00:57.065275+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:58.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: audit 2023-07-29T17:00:57.132992+0000 mgr.y (mgr.14144) 97 : audit [DBG] from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:58.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cephadm 2023-07-29T17:00:57.137126+0000 mgr.y (mgr.14144) 98 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cluster 2023-07-29T17:00:57.159995+0000 mon.a (mon.0) 297 : cluster [INF] osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819] boot 2023-07-29T17:00:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: cluster 2023-07-29T17:00:57.160067+0000 mon.a (mon.0) 298 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-07-29T17:00:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[28259]: audit 2023-07-29T17:00:57.160684+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:58.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cluster 2023-07-29T17:00:55.136938+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:58.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cluster 2023-07-29T17:00:55.137201+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:58.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cephadm 2023-07-29T17:00:56.740906+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:58.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cluster 2023-07-29T17:00:57.065275+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:58.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: audit 2023-07-29T17:00:57.132992+0000 mgr.y (mgr.14144) 97 : audit [DBG] from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:58.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cephadm 2023-07-29T17:00:57.137126+0000 mgr.y (mgr.14144) 98 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:58.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cluster 2023-07-29T17:00:57.159995+0000 mon.a (mon.0) 297 : cluster [INF] osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819] boot 2023-07-29T17:00:58.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: cluster 2023-07-29T17:00:57.160067+0000 mon.a (mon.0) 298 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-07-29T17:00:58.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:58 smithi078 bash[23704]: audit 2023-07-29T17:00:57.160684+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:00:58.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cluster 2023-07-29T17:00:55.136938+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:00:58.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cluster 2023-07-29T17:00:55.137201+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:00:58.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cephadm 2023-07-29T17:00:56.740906+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:00:58.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cluster 2023-07-29T17:00:57.065275+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-29T17:00:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: audit 2023-07-29T17:00:57.132992+0000 mgr.y (mgr.14144) 97 : audit [DBG] from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:00:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cephadm 2023-07-29T17:00:57.137126+0000 mgr.y (mgr.14144) 98 : cephadm [INF] Applying service osd.None on host smithi078... 2023-07-29T17:00:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cluster 2023-07-29T17:00:57.159995+0000 mon.a (mon.0) 297 : cluster [INF] osd.2 [v2:172.21.15.78:6818/3414832819,v1:172.21.15.78:6819/3414832819] boot 2023-07-29T17:00:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: cluster 2023-07-29T17:00:57.160067+0000 mon.a (mon.0) 298 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-07-29T17:00:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:58 smithi129 bash[23526]: audit 2023-07-29T17:00:57.160684+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:01:00.065 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:59 smithi078 bash[23704]: cluster 2023-07-29T17:00:58.768807+0000 mon.a (mon.0) 300 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-07-29T17:01:00.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:00:59 smithi078 bash[23704]: cluster 2023-07-29T17:00:59.065808+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:00.066 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:59 smithi078 bash[28259]: cluster 2023-07-29T17:00:58.768807+0000 mon.a (mon.0) 300 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-07-29T17:01:00.066 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:00:59 smithi078 bash[28259]: cluster 2023-07-29T17:00:59.065808+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:00.158 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:59 smithi129 bash[23526]: cluster 2023-07-29T17:00:58.768807+0000 mon.a (mon.0) 300 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-07-29T17:01:00.159 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:00:59 smithi129 bash[23526]: cluster 2023-07-29T17:00:59.065808+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:01.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:00 smithi129 bash[23526]: audit 2023-07-29T17:01:00.873476+0000 mon.a (mon.0) 301 : audit [INF] from='client.? 172.21.15.78:0/1877930427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eba2c490-d3d9-4b10-9d86-5cccb0bdad5b"}]: dispatch 2023-07-29T17:01:01.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:00 smithi129 bash[23526]: audit 2023-07-29T17:01:00.881916+0000 mon.a (mon.0) 302 : audit [INF] from='client.? 172.21.15.78:0/1877930427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eba2c490-d3d9-4b10-9d86-5cccb0bdad5b"}]': finished 2023-07-29T17:01:01.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:00 smithi129 bash[23526]: cluster 2023-07-29T17:01:00.882085+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-07-29T17:01:01.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:00 smithi129 bash[23526]: audit 2023-07-29T17:01:00.882356+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:01.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[23704]: audit 2023-07-29T17:01:00.873476+0000 mon.a (mon.0) 301 : audit [INF] from='client.? 172.21.15.78:0/1877930427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eba2c490-d3d9-4b10-9d86-5cccb0bdad5b"}]: dispatch 2023-07-29T17:01:01.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[23704]: audit 2023-07-29T17:01:00.881916+0000 mon.a (mon.0) 302 : audit [INF] from='client.? 172.21.15.78:0/1877930427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eba2c490-d3d9-4b10-9d86-5cccb0bdad5b"}]': finished 2023-07-29T17:01:01.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[23704]: cluster 2023-07-29T17:01:00.882085+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-07-29T17:01:01.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[23704]: audit 2023-07-29T17:01:00.882356+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:01.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[28259]: audit 2023-07-29T17:01:00.873476+0000 mon.a (mon.0) 301 : audit [INF] from='client.? 172.21.15.78:0/1877930427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eba2c490-d3d9-4b10-9d86-5cccb0bdad5b"}]: dispatch 2023-07-29T17:01:01.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[28259]: audit 2023-07-29T17:01:00.881916+0000 mon.a (mon.0) 302 : audit [INF] from='client.? 172.21.15.78:0/1877930427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eba2c490-d3d9-4b10-9d86-5cccb0bdad5b"}]': finished 2023-07-29T17:01:01.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[28259]: cluster 2023-07-29T17:01:00.882085+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-07-29T17:01:01.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:00 smithi078 bash[28259]: audit 2023-07-29T17:01:00.882356+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:02.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:02 smithi129 bash[23526]: cluster 2023-07-29T17:01:01.066331+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:02.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:02 smithi129 bash[23526]: audit 2023-07-29T17:01:01.532896+0000 mon.a (mon.0) 305 : audit [DBG] from='client.? 172.21.15.78:0/1497369591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:02.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:02 smithi078 bash[23704]: cluster 2023-07-29T17:01:01.066331+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:02.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:02 smithi078 bash[23704]: audit 2023-07-29T17:01:01.532896+0000 mon.a (mon.0) 305 : audit [DBG] from='client.? 172.21.15.78:0/1497369591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:02.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:02 smithi078 bash[28259]: cluster 2023-07-29T17:01:01.066331+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:02.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:02 smithi078 bash[28259]: audit 2023-07-29T17:01:01.532896+0000 mon.a (mon.0) 305 : audit [DBG] from='client.? 172.21.15.78:0/1497369591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:04.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:04 smithi129 bash[23526]: cluster 2023-07-29T17:01:03.066874+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:04.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:04 smithi078 bash[28259]: cluster 2023-07-29T17:01:03.066874+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:04.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:04 smithi078 bash[23704]: cluster 2023-07-29T17:01:03.066874+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:06.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:06 smithi129 bash[23526]: cluster 2023-07-29T17:01:05.067410+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:06.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:06 smithi078 bash[23704]: cluster 2023-07-29T17:01:05.067410+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:06.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:06 smithi078 bash[28259]: cluster 2023-07-29T17:01:05.067410+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:08.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:08 smithi129 bash[23526]: cluster 2023-07-29T17:01:07.067952+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:08.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:08 smithi078 bash[23704]: cluster 2023-07-29T17:01:07.067952+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:08.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:08 smithi078 bash[28259]: cluster 2023-07-29T17:01:07.067952+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:09.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:09 smithi129 bash[23526]: audit 2023-07-29T17:01:08.743698+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:01:09.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:09 smithi129 bash[23526]: audit 2023-07-29T17:01:08.745211+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:09.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:09 smithi078 bash[23704]: audit 2023-07-29T17:01:08.743698+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:01:09.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:09 smithi078 bash[23704]: audit 2023-07-29T17:01:08.745211+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:09.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:09 smithi078 bash[28259]: audit 2023-07-29T17:01:08.743698+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:01:09.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:09 smithi078 bash[28259]: audit 2023-07-29T17:01:08.745211+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:10.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:10 smithi078 bash[28259]: cephadm 2023-07-29T17:01:08.746384+0000 mgr.y (mgr.14144) 104 : cephadm [INF] Deploying daemon osd.3 on smithi078 2023-07-29T17:01:10.159 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:10 smithi078 bash[28259]: cluster 2023-07-29T17:01:09.068514+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:10.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:10 smithi078 bash[23704]: cephadm 2023-07-29T17:01:08.746384+0000 mgr.y (mgr.14144) 104 : cephadm [INF] Deploying daemon osd.3 on smithi078 2023-07-29T17:01:10.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:10 smithi078 bash[23704]: cluster 2023-07-29T17:01:09.068514+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:10.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:10 smithi129 bash[23526]: cephadm 2023-07-29T17:01:08.746384+0000 mgr.y (mgr.14144) 104 : cephadm [INF] Deploying daemon osd.3 on smithi078 2023-07-29T17:01:10.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:10 smithi129 bash[23526]: cluster 2023-07-29T17:01:09.068514+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:11.215 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2023-07-29T17:01:12.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:12 smithi129 bash[23526]: cluster 2023-07-29T17:01:11.068994+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:12.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:12 smithi129 bash[23526]: audit 2023-07-29T17:01:11.194655+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:12.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:12 smithi129 bash[23526]: audit 2023-07-29T17:01:11.199985+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:12.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:12 smithi129 bash[23526]: audit 2023-07-29T17:01:11.207059+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:12.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:12 smithi129 bash[23526]: audit 2023-07-29T17:01:11.209147+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-07-29T17:01:12.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[23704]: cluster 2023-07-29T17:01:11.068994+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:12.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[23704]: audit 2023-07-29T17:01:11.194655+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:12.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[23704]: audit 2023-07-29T17:01:11.199985+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:12.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[23704]: audit 2023-07-29T17:01:11.207059+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:12.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[23704]: audit 2023-07-29T17:01:11.209147+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-07-29T17:01:12.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[28259]: cluster 2023-07-29T17:01:11.068994+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:12.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[28259]: audit 2023-07-29T17:01:11.194655+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:12.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[28259]: audit 2023-07-29T17:01:11.199985+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:12.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[28259]: audit 2023-07-29T17:01:11.207059+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:12.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:12 smithi078 bash[28259]: audit 2023-07-29T17:01:11.209147+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-07-29T17:01:12.523 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.3.service 2023-07-29T17:01:12.526 INFO:tasks.cephadm:Deploying osd.4 on smithi129 with /dev/vg_nvme/lv_4... 2023-07-29T17:01:12.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-29T17:01:12.774 INFO:journalctl@ceph.osd.3.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:01:13.425 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-29T17:01:13.426 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2023-07-29T17:01:13.426 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2023-07-29T17:01:13.426 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2023-07-29T17:01:13.426 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-29T17:01:13.426 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-29T17:01:13.426 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[39030]: --> ceph-volume lvm activate successful for osd ID: 3 2023-07-29T17:01:13.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[23704]: audit 2023-07-29T17:01:12.212509+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-07-29T17:01:13.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[23704]: cluster 2023-07-29T17:01:12.212624+0000 mon.a (mon.0) 313 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-07-29T17:01:13.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[23704]: audit 2023-07-29T17:01:12.212824+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:13.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[23704]: audit 2023-07-29T17:01:12.214673+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-07-29T17:01:13.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[23704]: cluster 2023-07-29T17:01:13.069442+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:13.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[28259]: audit 2023-07-29T17:01:12.212509+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-07-29T17:01:13.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[28259]: cluster 2023-07-29T17:01:12.212624+0000 mon.a (mon.0) 313 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-07-29T17:01:13.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[28259]: audit 2023-07-29T17:01:12.212824+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:13.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[28259]: audit 2023-07-29T17:01:12.214673+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-07-29T17:01:13.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:13 smithi078 bash[28259]: cluster 2023-07-29T17:01:13.069442+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:13.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:13 smithi129 bash[23526]: audit 2023-07-29T17:01:12.212509+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-07-29T17:01:13.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:13 smithi129 bash[23526]: cluster 2023-07-29T17:01:12.212624+0000 mon.a (mon.0) 313 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-07-29T17:01:13.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:13 smithi129 bash[23526]: audit 2023-07-29T17:01:12.212824+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:13.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:13 smithi129 bash[23526]: audit 2023-07-29T17:01:12.214673+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-07-29T17:01:13.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:13 smithi129 bash[23526]: cluster 2023-07-29T17:01:13.069442+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:14.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:14 smithi129 bash[23526]: audit 2023-07-29T17:01:13.222335+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-07-29T17:01:14.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:14 smithi129 bash[23526]: cluster 2023-07-29T17:01:13.222537+0000 mon.a (mon.0) 317 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-07-29T17:01:14.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:14 smithi129 bash[23526]: audit 2023-07-29T17:01:13.223264+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:14.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[28259]: audit 2023-07-29T17:01:13.222335+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-07-29T17:01:14.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[28259]: cluster 2023-07-29T17:01:13.222537+0000 mon.a (mon.0) 317 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-07-29T17:01:14.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[28259]: audit 2023-07-29T17:01:13.223264+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:14.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[23704]: audit 2023-07-29T17:01:13.222335+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-07-29T17:01:14.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[23704]: cluster 2023-07-29T17:01:13.222537+0000 mon.a (mon.0) 317 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-07-29T17:01:14.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[23704]: audit 2023-07-29T17:01:13.223264+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:15.176 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:14 smithi078 bash[39179]: debug 2023-07-29T17:01:14.908+0000 7fa3a0d81700 1 -- 172.21.15.78:0/510616718 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5603dcfc44e0 con 0x5603dd03d400 2023-07-29T17:01:15.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[28259]: cluster 2023-07-29T17:01:14.224463+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-07-29T17:01:15.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[28259]: audit 2023-07-29T17:01:14.225246+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:15.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[28259]: audit 2023-07-29T17:01:14.990412+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:15.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[28259]: cluster 2023-07-29T17:01:15.069914+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:15.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[23704]: cluster 2023-07-29T17:01:14.224463+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-07-29T17:01:15.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[23704]: audit 2023-07-29T17:01:14.225246+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:15.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[23704]: audit 2023-07-29T17:01:14.990412+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:15.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[23704]: cluster 2023-07-29T17:01:15.069914+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:15.564 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:15 smithi129 bash[23526]: cluster 2023-07-29T17:01:14.224463+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-07-29T17:01:15.564 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:15 smithi129 bash[23526]: audit 2023-07-29T17:01:14.225246+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:15.565 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:15 smithi129 bash[23526]: audit 2023-07-29T17:01:14.990412+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:15.565 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:15 smithi129 bash[23526]: cluster 2023-07-29T17:01:15.069914+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:15.925 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:15 smithi078 bash[39179]: debug 2023-07-29T17:01:15.452+0000 7fa3a880c080 -1 Falling back to public interface 2023-07-29T17:01:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-29T17:01:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-29T17:01:16.185 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2023-07-29T17:01:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2023-07-29T17:01:16.185 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0205298 s, 511 MB/s 2023-07-29T17:01:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2023-07-29T17:01:16.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2023-07-29T17:01:18.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:18 smithi078 bash[23704]: cluster 2023-07-29T17:01:17.070552+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:18.175 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:18 smithi078 bash[28259]: cluster 2023-07-29T17:01:17.070552+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:18.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:18 smithi129 bash[23526]: cluster 2023-07-29T17:01:17.070552+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:18.925 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:18 smithi078 bash[39179]: debug 2023-07-29T17:01:18.476+0000 7fa3a880c080 -1 osd.3 0 log_to_monitors {default=true} 2023-07-29T17:01:19.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:19 smithi078 bash[23704]: audit 2023-07-29T17:01:18.487107+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:01:19.358 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:19 smithi078 bash[23704]: audit 2023-07-29T17:01:18.487538+0000 mon.a (mon.0) 322 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:01:19.358 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:19 smithi078 bash[28259]: audit 2023-07-29T17:01:18.487107+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:01:19.358 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:19 smithi078 bash[28259]: audit 2023-07-29T17:01:18.487538+0000 mon.a (mon.0) 322 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:01:19.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:19 smithi129 bash[23526]: audit 2023-07-29T17:01:18.487107+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:01:19.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:19 smithi129 bash[23526]: audit 2023-07-29T17:01:18.487538+0000 mon.a (mon.0) 322 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:01:20.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: cluster 2023-07-29T17:01:19.071099+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.100670+0000 mon.a (mon.0) 323 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T17:01:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: cluster 2023-07-29T17:01:19.100806+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-07-29T17:01:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.101081+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.101859+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:01:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.102340+0000 mon.a (mon.0) 326 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:01:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.855676+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:01:20.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.860128+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:01:20.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:19.861904+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:20.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:20 smithi129 bash[23526]: audit 2023-07-29T17:01:20.094662+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:20.425 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[39179]: debug 2023-07-29T17:01:20.104+0000 7fa39ed7d700 -1 osd.3 0 waiting for initial osdmap 2023-07-29T17:01:20.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: cluster 2023-07-29T17:01:19.071099+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:20.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.100670+0000 mon.a (mon.0) 323 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T17:01:20.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: cluster 2023-07-29T17:01:19.100806+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-07-29T17:01:20.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.101081+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:20.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.101859+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:01:20.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.102340+0000 mon.a (mon.0) 326 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:01:20.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.855676+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:01:20.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.860128+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:01:20.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:19.861904+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:20.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[23704]: audit 2023-07-29T17:01:20.094662+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:20.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: cluster 2023-07-29T17:01:19.071099+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:20.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.100670+0000 mon.a (mon.0) 323 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: cluster 2023-07-29T17:01:19.100806+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.101081+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.101859+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.102340+0000 mon.a (mon.0) 326 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.855676+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.860128+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:01:20.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:19.861904+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:20.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:20 smithi078 bash[28259]: audit 2023-07-29T17:01:20.094662+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:21.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:19.853841+0000 mgr.y (mgr.14144) 111 : audit [DBG] from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:01:21.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: cephadm 2023-07-29T17:01:19.859544+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:01:21.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: cephadm 2023-07-29T17:01:20.086708+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:01:21.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:20.102846+0000 mon.a (mon.0) 331 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:01:21.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: cluster 2023-07-29T17:01:20.102960+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2023-07-29T17:01:21.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:20.103679+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:21.376 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:20.110972+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:21.376 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:20.328335+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:21.376 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:20.331407+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:21.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:21 smithi129 bash[23526]: audit 2023-07-29T17:01:20.340845+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:21.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:19.853841+0000 mgr.y (mgr.14144) 111 : audit [DBG] from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:01:21.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: cephadm 2023-07-29T17:01:19.859544+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:01:21.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: cephadm 2023-07-29T17:01:20.086708+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:01:21.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:20.102846+0000 mon.a (mon.0) 331 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:01:21.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: cluster 2023-07-29T17:01:20.102960+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2023-07-29T17:01:21.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:20.103679+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:21.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:20.110972+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:21.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:20.328335+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:21.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:20.331407+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:21.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[28259]: audit 2023-07-29T17:01:20.340845+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:21.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:19.853841+0000 mgr.y (mgr.14144) 111 : audit [DBG] from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:01:21.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: cephadm 2023-07-29T17:01:19.859544+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:01:21.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: cephadm 2023-07-29T17:01:20.086708+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Detected new or changed devices on smithi078 2023-07-29T17:01:21.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:20.102846+0000 mon.a (mon.0) 331 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-29T17:01:21.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: cluster 2023-07-29T17:01:20.102960+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2023-07-29T17:01:21.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:20.103679+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:21.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:20.110972+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:21.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:20.328335+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:21.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:20.331407+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:21.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:21 smithi078 bash[23704]: audit 2023-07-29T17:01:20.340845+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:22.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: cluster 2023-07-29T17:01:19.513759+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:01:22.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: cluster 2023-07-29T17:01:19.513963+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:01:22.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: cluster 2023-07-29T17:01:21.071664+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:22.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: audit 2023-07-29T17:01:21.108292+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:22.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: cluster 2023-07-29T17:01:21.117681+0000 mon.a (mon.0) 339 : cluster [INF] osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573] boot 2023-07-29T17:01:22.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: cluster 2023-07-29T17:01:21.117797+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-07-29T17:01:22.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:22 smithi129 bash[23526]: audit 2023-07-29T17:01:21.118590+0000 mon.a (mon.0) 341 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:22.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: cluster 2023-07-29T17:01:19.513759+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:01:22.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: cluster 2023-07-29T17:01:19.513963+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:01:22.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: cluster 2023-07-29T17:01:21.071664+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:22.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: audit 2023-07-29T17:01:21.108292+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:22.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: cluster 2023-07-29T17:01:21.117681+0000 mon.a (mon.0) 339 : cluster [INF] osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573] boot 2023-07-29T17:01:22.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: cluster 2023-07-29T17:01:21.117797+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-07-29T17:01:22.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[28259]: audit 2023-07-29T17:01:21.118590+0000 mon.a (mon.0) 341 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:22.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: cluster 2023-07-29T17:01:19.513759+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:01:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: cluster 2023-07-29T17:01:19.513963+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:01:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: cluster 2023-07-29T17:01:21.071664+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-29T17:01:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: audit 2023-07-29T17:01:21.108292+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: cluster 2023-07-29T17:01:21.117681+0000 mon.a (mon.0) 339 : cluster [INF] osd.3 [v2:172.21.15.78:6826/4175854573,v1:172.21.15.78:6827/4175854573] boot 2023-07-29T17:01:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: cluster 2023-07-29T17:01:21.117797+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-07-29T17:01:22.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:22 smithi078 bash[23704]: audit 2023-07-29T17:01:21.118590+0000 mon.a (mon.0) 341 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:01:23.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:23 smithi129 bash[23526]: cluster 2023-07-29T17:01:22.347266+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-07-29T17:01:23.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:23 smithi129 bash[23526]: cluster 2023-07-29T17:01:23.072160+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:23.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:23 smithi078 bash[23704]: cluster 2023-07-29T17:01:22.347266+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-07-29T17:01:23.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:23 smithi078 bash[23704]: cluster 2023-07-29T17:01:23.072160+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:23.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:23 smithi078 bash[28259]: cluster 2023-07-29T17:01:22.347266+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-07-29T17:01:23.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:23 smithi078 bash[28259]: cluster 2023-07-29T17:01:23.072160+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:24.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:24 smithi129 bash[23526]: audit 2023-07-29T17:01:23.588873+0000 mon.b (mon.1) 8 : audit [INF] from='client.? 172.21.15.129:0/334897652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]: dispatch 2023-07-29T17:01:24.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:24 smithi129 bash[23526]: audit 2023-07-29T17:01:23.589387+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]: dispatch 2023-07-29T17:01:24.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:24 smithi129 bash[23526]: audit 2023-07-29T17:01:23.597922+0000 mon.a (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]': finished 2023-07-29T17:01:24.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:24 smithi129 bash[23526]: cluster 2023-07-29T17:01:23.598084+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-07-29T17:01:24.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:24 smithi129 bash[23526]: audit 2023-07-29T17:01:23.598293+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:24.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:24 smithi129 bash[23526]: audit 2023-07-29T17:01:24.248307+0000 mon.b (mon.1) 9 : audit [DBG] from='client.? 172.21.15.129:0/1110650978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:24.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[23704]: audit 2023-07-29T17:01:23.588873+0000 mon.b (mon.1) 8 : audit [INF] from='client.? 172.21.15.129:0/334897652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]: dispatch 2023-07-29T17:01:24.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[23704]: audit 2023-07-29T17:01:23.589387+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]: dispatch 2023-07-29T17:01:24.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[23704]: audit 2023-07-29T17:01:23.597922+0000 mon.a (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]': finished 2023-07-29T17:01:24.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[23704]: cluster 2023-07-29T17:01:23.598084+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-07-29T17:01:24.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[23704]: audit 2023-07-29T17:01:23.598293+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:24.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[23704]: audit 2023-07-29T17:01:24.248307+0000 mon.b (mon.1) 9 : audit [DBG] from='client.? 172.21.15.129:0/1110650978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:24.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[28259]: audit 2023-07-29T17:01:23.588873+0000 mon.b (mon.1) 8 : audit [INF] from='client.? 172.21.15.129:0/334897652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]: dispatch 2023-07-29T17:01:24.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[28259]: audit 2023-07-29T17:01:23.589387+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]: dispatch 2023-07-29T17:01:24.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[28259]: audit 2023-07-29T17:01:23.597922+0000 mon.a (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8"}]': finished 2023-07-29T17:01:24.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[28259]: cluster 2023-07-29T17:01:23.598084+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-07-29T17:01:24.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[28259]: audit 2023-07-29T17:01:23.598293+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:24.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:24 smithi078 bash[28259]: audit 2023-07-29T17:01:24.248307+0000 mon.b (mon.1) 9 : audit [DBG] from='client.? 172.21.15.129:0/1110650978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:25.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:25 smithi129 bash[23526]: cluster 2023-07-29T17:01:25.072865+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:25.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:25 smithi078 bash[23704]: cluster 2023-07-29T17:01:25.072865+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:25.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:25 smithi078 bash[28259]: cluster 2023-07-29T17:01:25.072865+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:28.348 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:28 smithi129 bash[23526]: cluster 2023-07-29T17:01:27.073559+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:28.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:28 smithi078 bash[23704]: cluster 2023-07-29T17:01:27.073559+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:28.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:28 smithi078 bash[28259]: cluster 2023-07-29T17:01:27.073559+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:30.158 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:30 smithi129 bash[23526]: cluster 2023-07-29T17:01:29.074151+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:30.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:30 smithi078 bash[23704]: cluster 2023-07-29T17:01:29.074151+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:30.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:30 smithi078 bash[28259]: cluster 2023-07-29T17:01:29.074151+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:31.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:31 smithi129 bash[23526]: audit 2023-07-29T17:01:30.923180+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:01:31.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:31 smithi129 bash[23526]: audit 2023-07-29T17:01:30.925164+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:31.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:31 smithi078 bash[23704]: audit 2023-07-29T17:01:30.923180+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:01:31.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:31 smithi078 bash[23704]: audit 2023-07-29T17:01:30.925164+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:31.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:31 smithi078 bash[28259]: audit 2023-07-29T17:01:30.923180+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:01:31.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:31 smithi078 bash[28259]: audit 2023-07-29T17:01:30.925164+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:32.339 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:32 smithi129 bash[23526]: cephadm 2023-07-29T17:01:30.926727+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Deploying daemon osd.4 on smithi129 2023-07-29T17:01:32.339 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:32 smithi129 bash[23526]: cluster 2023-07-29T17:01:31.074696+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:32.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:32 smithi078 bash[28259]: cephadm 2023-07-29T17:01:30.926727+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Deploying daemon osd.4 on smithi129 2023-07-29T17:01:32.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:32 smithi078 bash[28259]: cluster 2023-07-29T17:01:31.074696+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:32.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:32 smithi078 bash[23704]: cephadm 2023-07-29T17:01:30.926727+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Deploying daemon osd.4 on smithi129 2023-07-29T17:01:32.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:32 smithi078 bash[23704]: cluster 2023-07-29T17:01:31.074696+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:33.298 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 4 on host 'smithi129' 2023-07-29T17:01:34.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:34 smithi129 bash[23526]: cluster 2023-07-29T17:01:33.075358+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:34.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:34 smithi129 bash[23526]: audit 2023-07-29T17:01:33.274149+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:34.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:34 smithi129 bash[23526]: audit 2023-07-29T17:01:33.283157+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:34.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:34 smithi129 bash[23526]: audit 2023-07-29T17:01:33.293871+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:34.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[28259]: cluster 2023-07-29T17:01:33.075358+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:34.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[28259]: audit 2023-07-29T17:01:33.274149+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:34.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[28259]: audit 2023-07-29T17:01:33.283157+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:34.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[28259]: audit 2023-07-29T17:01:33.293871+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:34.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[23704]: cluster 2023-07-29T17:01:33.075358+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:34.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[23704]: audit 2023-07-29T17:01:33.274149+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:34.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[23704]: audit 2023-07-29T17:01:33.283157+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:34.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:34 smithi078 bash[23704]: audit 2023-07-29T17:01:33.293871+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:34.472 DEBUG:teuthology.orchestra.run.smithi129:osd.4> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.4.service 2023-07-29T17:01:34.475 INFO:tasks.cephadm:Deploying osd.5 on smithi129 with /dev/vg_nvme/lv_3... 2023-07-29T17:01:34.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-29T17:01:35.286 INFO:journalctl@ceph.osd.4.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:01:35.788 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-29T17:01:35.789 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2023-07-29T17:01:35.789 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2023-07-29T17:01:35.789 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2023-07-29T17:01:35.789 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-29T17:01:35.789 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-29T17:01:35.789 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:35 smithi129 bash[26503]: --> ceph-volume lvm activate successful for osd ID: 4 2023-07-29T17:01:36.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:36 smithi129 bash[23526]: cluster 2023-07-29T17:01:35.076011+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:36.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:36 smithi129 bash[23526]: audit 2023-07-29T17:01:35.243226+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:01:36.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:36 smithi129 bash[23526]: audit 2023-07-29T17:01:35.245617+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:01:36.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:36 smithi078 bash[28259]: cluster 2023-07-29T17:01:35.076011+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:36.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:36 smithi078 bash[28259]: audit 2023-07-29T17:01:35.243226+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:01:36.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:36 smithi078 bash[28259]: audit 2023-07-29T17:01:35.245617+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:01:36.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:36 smithi078 bash[23704]: cluster 2023-07-29T17:01:35.076011+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:36.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:36 smithi078 bash[23704]: audit 2023-07-29T17:01:35.243226+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:01:36.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:36 smithi078 bash[23704]: audit 2023-07-29T17:01:35.245617+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:01:38.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:38 smithi129 bash[23526]: cluster 2023-07-29T17:01:37.076574+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:38.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:38 smithi129 bash[23526]: audit 2023-07-29T17:01:37.184573+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:38.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:38 smithi078 bash[23704]: cluster 2023-07-29T17:01:37.076574+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:38.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:38 smithi078 bash[23704]: audit 2023-07-29T17:01:37.184573+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:38.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:38 smithi078 bash[28259]: cluster 2023-07-29T17:01:37.076574+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:38.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:38 smithi078 bash[28259]: audit 2023-07-29T17:01:37.184573+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:39.159 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:38 smithi129 bash[26703]: debug 2023-07-29T17:01:38.737+0000 7f3f50eb8700 1 -- 172.21.15.129:0/1293708630 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5555d81204e0 con 0x5555d8199000 2023-07-29T17:01:39.537 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:39 smithi129 bash[26703]: debug 2023-07-29T17:01:39.301+0000 7f3f58943080 -1 Falling back to public interface 2023-07-29T17:01:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-29T17:01:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-29T17:01:39.858 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2023-07-29T17:01:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2023-07-29T17:01:39.859 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.016988 s, 617 MB/s 2023-07-29T17:01:39.859 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2023-07-29T17:01:40.161 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:40 smithi129 bash[23526]: cluster 2023-07-29T17:01:39.077113+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:40.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:40 smithi078 bash[23704]: cluster 2023-07-29T17:01:39.077113+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:40.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:40 smithi078 bash[28259]: cluster 2023-07-29T17:01:39.077113+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:40.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2023-07-29T17:01:42.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:42 smithi129 bash[23526]: cluster 2023-07-29T17:01:41.077642+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:42.409 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:42 smithi129 bash[26703]: debug 2023-07-29T17:01:42.289+0000 7f3f58943080 -1 osd.4 0 log_to_monitors {default=true} 2023-07-29T17:01:42.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:42 smithi078 bash[23704]: cluster 2023-07-29T17:01:41.077642+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:42.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:42 smithi078 bash[28259]: cluster 2023-07-29T17:01:41.077642+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:43.355 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:43 smithi129 bash[23526]: audit 2023-07-29T17:01:42.299391+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:01:43.355 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:43 smithi129 bash[23526]: audit 2023-07-29T17:01:42.300442+0000 mon.a (mon.0) 355 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:01:43.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:43 smithi078 bash[23704]: audit 2023-07-29T17:01:42.299391+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:01:43.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:43 smithi078 bash[23704]: audit 2023-07-29T17:01:42.300442+0000 mon.a (mon.0) 355 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:01:43.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:43 smithi078 bash[28259]: audit 2023-07-29T17:01:42.299391+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:01:43.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:43 smithi078 bash[28259]: audit 2023-07-29T17:01:42.300442+0000 mon.a (mon.0) 355 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:01:44.409 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[26703]: debug 2023-07-29T17:01:44.157+0000 7f3f4eeb4700 -1 osd.4 0 waiting for initial osdmap 2023-07-29T17:01:44.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[23526]: cluster 2023-07-29T17:01:43.078207+0000 mgr.y (mgr.14144) 126 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:44.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[23526]: audit 2023-07-29T17:01:43.146466+0000 mon.a (mon.0) 356 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T17:01:44.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[23526]: cluster 2023-07-29T17:01:43.146649+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-07-29T17:01:44.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[23526]: audit 2023-07-29T17:01:43.146822+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:44.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[23526]: audit 2023-07-29T17:01:43.147858+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:01:44.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:44 smithi129 bash[23526]: audit 2023-07-29T17:01:43.148830+0000 mon.a (mon.0) 359 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:01:44.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[23704]: cluster 2023-07-29T17:01:43.078207+0000 mgr.y (mgr.14144) 126 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:44.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[23704]: audit 2023-07-29T17:01:43.146466+0000 mon.a (mon.0) 356 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T17:01:44.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[23704]: cluster 2023-07-29T17:01:43.146649+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-07-29T17:01:44.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[23704]: audit 2023-07-29T17:01:43.146822+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:44.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[23704]: audit 2023-07-29T17:01:43.147858+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:01:44.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[23704]: audit 2023-07-29T17:01:43.148830+0000 mon.a (mon.0) 359 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:01:44.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[28259]: cluster 2023-07-29T17:01:43.078207+0000 mgr.y (mgr.14144) 126 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:44.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[28259]: audit 2023-07-29T17:01:43.146466+0000 mon.a (mon.0) 356 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T17:01:44.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[28259]: cluster 2023-07-29T17:01:43.146649+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-07-29T17:01:44.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[28259]: audit 2023-07-29T17:01:43.146822+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:44.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[28259]: audit 2023-07-29T17:01:43.147858+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:01:44.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:44 smithi078 bash[28259]: audit 2023-07-29T17:01:43.148830+0000 mon.a (mon.0) 359 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:01:45.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.150946+0000 mon.a (mon.0) 360 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:01:45.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: cluster 2023-07-29T17:01:44.151201+0000 mon.a (mon.0) 361 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-07-29T17:01:45.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.152039+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.159810+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.512100+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.520271+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.523094+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.530716+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.752046+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:01:45.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.755397+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:01:45.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:44.757013+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:45.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: cluster 2023-07-29T17:01:45.151820+0000 mon.a (mon.0) 371 : cluster [INF] osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569] boot 2023-07-29T17:01:45.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: cluster 2023-07-29T17:01:45.151978+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-07-29T17:01:45.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:45 smithi129 bash[23526]: audit 2023-07-29T17:01:45.152311+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.150946+0000 mon.a (mon.0) 360 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:01:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: cluster 2023-07-29T17:01:44.151201+0000 mon.a (mon.0) 361 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-07-29T17:01:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.152039+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.159810+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.512100+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.520271+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.523094+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.530716+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.752046+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.755397+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:44.757013+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: cluster 2023-07-29T17:01:45.151820+0000 mon.a (mon.0) 371 : cluster [INF] osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569] boot 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: cluster 2023-07-29T17:01:45.151978+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-07-29T17:01:45.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[23704]: audit 2023-07-29T17:01:45.152311+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.150946+0000 mon.a (mon.0) 360 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:01:45.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: cluster 2023-07-29T17:01:44.151201+0000 mon.a (mon.0) 361 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-07-29T17:01:45.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.152039+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.159810+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:45.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.512100+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.520271+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.523094+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.530716+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.752046+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.755397+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:44.757013+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:45.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: cluster 2023-07-29T17:01:45.151820+0000 mon.a (mon.0) 371 : cluster [INF] osd.4 [v2:172.21.15.129:6800/1329658569,v1:172.21.15.129:6801/1329658569] boot 2023-07-29T17:01:45.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: cluster 2023-07-29T17:01:45.151978+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-07-29T17:01:45.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:45 smithi078 bash[28259]: audit 2023-07-29T17:01:45.152311+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:01:46.254 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: cluster 2023-07-29T17:01:43.252032+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:01:46.254 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: cluster 2023-07-29T17:01:43.252254+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:01:46.254 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: cephadm 2023-07-29T17:01:44.503128+0000 mgr.y (mgr.14144) 127 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:01:46.254 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: audit 2023-07-29T17:01:44.750396+0000 mgr.y (mgr.14144) 128 : audit [DBG] from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:01:46.254 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: cephadm 2023-07-29T17:01:44.754841+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:01:46.255 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: cluster 2023-07-29T17:01:45.078737+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:46.255 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:46 smithi129 bash[23526]: cluster 2023-07-29T17:01:46.093504+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-07-29T17:01:46.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: cluster 2023-07-29T17:01:43.252032+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:01:46.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: cluster 2023-07-29T17:01:43.252254+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:01:46.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: cephadm 2023-07-29T17:01:44.503128+0000 mgr.y (mgr.14144) 127 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:01:46.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: audit 2023-07-29T17:01:44.750396+0000 mgr.y (mgr.14144) 128 : audit [DBG] from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:01:46.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: cephadm 2023-07-29T17:01:44.754841+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:01:46.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: cluster 2023-07-29T17:01:45.078737+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:46.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[23704]: cluster 2023-07-29T17:01:46.093504+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-07-29T17:01:46.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: cluster 2023-07-29T17:01:43.252032+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:01:46.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: cluster 2023-07-29T17:01:43.252254+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:01:46.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: cephadm 2023-07-29T17:01:44.503128+0000 mgr.y (mgr.14144) 127 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:01:46.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: audit 2023-07-29T17:01:44.750396+0000 mgr.y (mgr.14144) 128 : audit [DBG] from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:01:46.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: cephadm 2023-07-29T17:01:44.754841+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:01:46.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: cluster 2023-07-29T17:01:45.078737+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-29T17:01:46.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:46 smithi078 bash[28259]: cluster 2023-07-29T17:01:46.093504+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-07-29T17:01:48.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:48 smithi129 bash[23526]: cluster 2023-07-29T17:01:47.079313+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v118: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:48.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:48 smithi129 bash[23526]: cluster 2023-07-29T17:01:47.096640+0000 mon.a (mon.0) 375 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-07-29T17:01:48.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:48 smithi078 bash[23704]: cluster 2023-07-29T17:01:47.079313+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v118: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:48.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:48 smithi078 bash[23704]: cluster 2023-07-29T17:01:47.096640+0000 mon.a (mon.0) 375 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-07-29T17:01:48.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:48 smithi078 bash[28259]: cluster 2023-07-29T17:01:47.079313+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v118: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:48.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:48 smithi078 bash[28259]: cluster 2023-07-29T17:01:47.096640+0000 mon.a (mon.0) 375 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-07-29T17:01:49.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:49 smithi129 bash[23526]: audit 2023-07-29T17:01:48.482668+0000 mon.b (mon.1) 12 : audit [INF] from='client.? 172.21.15.129:0/2703167696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]: dispatch 2023-07-29T17:01:49.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:49 smithi129 bash[23526]: audit 2023-07-29T17:01:48.483828+0000 mon.a (mon.0) 376 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]: dispatch 2023-07-29T17:01:49.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:49 smithi129 bash[23526]: audit 2023-07-29T17:01:48.497133+0000 mon.a (mon.0) 377 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]': finished 2023-07-29T17:01:49.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:49 smithi129 bash[23526]: cluster 2023-07-29T17:01:48.497430+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-07-29T17:01:49.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:49 smithi129 bash[23526]: audit 2023-07-29T17:01:48.497711+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:01:49.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[23704]: audit 2023-07-29T17:01:48.482668+0000 mon.b (mon.1) 12 : audit [INF] from='client.? 172.21.15.129:0/2703167696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]: dispatch 2023-07-29T17:01:49.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[23704]: audit 2023-07-29T17:01:48.483828+0000 mon.a (mon.0) 376 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]: dispatch 2023-07-29T17:01:49.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[23704]: audit 2023-07-29T17:01:48.497133+0000 mon.a (mon.0) 377 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]': finished 2023-07-29T17:01:49.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[23704]: cluster 2023-07-29T17:01:48.497430+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-07-29T17:01:49.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[23704]: audit 2023-07-29T17:01:48.497711+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:01:49.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[28259]: audit 2023-07-29T17:01:48.482668+0000 mon.b (mon.1) 12 : audit [INF] from='client.? 172.21.15.129:0/2703167696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]: dispatch 2023-07-29T17:01:49.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[28259]: audit 2023-07-29T17:01:48.483828+0000 mon.a (mon.0) 376 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]: dispatch 2023-07-29T17:01:49.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[28259]: audit 2023-07-29T17:01:48.497133+0000 mon.a (mon.0) 377 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd04f2c7-9386-4d16-9f41-bd785f3eab89"}]': finished 2023-07-29T17:01:49.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[28259]: cluster 2023-07-29T17:01:48.497430+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-07-29T17:01:49.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:49 smithi078 bash[28259]: audit 2023-07-29T17:01:48.497711+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:01:50.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:50 smithi129 bash[23526]: cluster 2023-07-29T17:01:49.079865+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v121: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:50.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:50 smithi129 bash[23526]: audit 2023-07-29T17:01:49.166907+0000 mon.b (mon.1) 13 : audit [DBG] from='client.? 172.21.15.129:0/4276436838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:50.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:50 smithi078 bash[23704]: cluster 2023-07-29T17:01:49.079865+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v121: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:50.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:50 smithi078 bash[23704]: audit 2023-07-29T17:01:49.166907+0000 mon.b (mon.1) 13 : audit [DBG] from='client.? 172.21.15.129:0/4276436838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:50.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:50 smithi078 bash[28259]: cluster 2023-07-29T17:01:49.079865+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v121: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:50.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:50 smithi078 bash[28259]: audit 2023-07-29T17:01:49.166907+0000 mon.b (mon.1) 13 : audit [DBG] from='client.? 172.21.15.129:0/4276436838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:01:52.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:52 smithi129 bash[23526]: cluster 2023-07-29T17:01:51.080417+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v122: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:52.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:52 smithi078 bash[28259]: cluster 2023-07-29T17:01:51.080417+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v122: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:52.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:52 smithi078 bash[23704]: cluster 2023-07-29T17:01:51.080417+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v122: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:54.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:54 smithi129 bash[23526]: cluster 2023-07-29T17:01:53.080992+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:54.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:54 smithi078 bash[28259]: cluster 2023-07-29T17:01:53.080992+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:54.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:54 smithi078 bash[23704]: cluster 2023-07-29T17:01:53.080992+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:56 smithi129 bash[23526]: cluster 2023-07-29T17:01:55.081569+0000 mgr.y (mgr.14144) 135 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:56 smithi129 bash[23526]: audit 2023-07-29T17:01:56.081281+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:01:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:56 smithi129 bash[23526]: audit 2023-07-29T17:01:56.083063+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:56.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:56 smithi078 bash[23704]: cluster 2023-07-29T17:01:55.081569+0000 mgr.y (mgr.14144) 135 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:56.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:56 smithi078 bash[23704]: audit 2023-07-29T17:01:56.081281+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:01:56.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:56 smithi078 bash[23704]: audit 2023-07-29T17:01:56.083063+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:56.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:56 smithi078 bash[28259]: cluster 2023-07-29T17:01:55.081569+0000 mgr.y (mgr.14144) 135 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:56.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:56 smithi078 bash[28259]: audit 2023-07-29T17:01:56.081281+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:01:56.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:56 smithi078 bash[28259]: audit 2023-07-29T17:01:56.083063+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:01:57.405 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:57 smithi129 bash[23526]: cephadm 2023-07-29T17:01:56.084701+0000 mgr.y (mgr.14144) 136 : cephadm [INF] Deploying daemon osd.5 on smithi129 2023-07-29T17:01:57.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:57 smithi078 bash[28259]: cephadm 2023-07-29T17:01:56.084701+0000 mgr.y (mgr.14144) 136 : cephadm [INF] Deploying daemon osd.5 on smithi129 2023-07-29T17:01:57.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:57 smithi078 bash[23704]: cephadm 2023-07-29T17:01:56.084701+0000 mgr.y (mgr.14144) 136 : cephadm [INF] Deploying daemon osd.5 on smithi129 2023-07-29T17:01:58.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:58 smithi129 bash[23526]: cluster 2023-07-29T17:01:57.082154+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:58.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:58 smithi078 bash[23704]: cluster 2023-07-29T17:01:57.082154+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:58 smithi078 bash[28259]: cluster 2023-07-29T17:01:57.082154+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:01:58.565 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 5 on host 'smithi129' 2023-07-29T17:01:59.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:59 smithi129 bash[23526]: audit 2023-07-29T17:01:58.542805+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:59.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:59 smithi129 bash[23526]: audit 2023-07-29T17:01:58.551063+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:59.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:01:59 smithi129 bash[23526]: audit 2023-07-29T17:01:58.560284+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:59.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:59 smithi078 bash[23704]: audit 2023-07-29T17:01:58.542805+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:59.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:59 smithi078 bash[23704]: audit 2023-07-29T17:01:58.551063+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:59.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:01:59 smithi078 bash[23704]: audit 2023-07-29T17:01:58.560284+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:59.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:59 smithi078 bash[28259]: audit 2023-07-29T17:01:58.542805+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:01:59.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:59 smithi078 bash[28259]: audit 2023-07-29T17:01:58.551063+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:59.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:01:59 smithi078 bash[28259]: audit 2023-07-29T17:01:58.560284+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:01:59.808 DEBUG:teuthology.orchestra.run.smithi129:osd.5> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.5.service 2023-07-29T17:01:59.810 INFO:tasks.cephadm:Deploying osd.6 on smithi129 with /dev/vg_nvme/lv_2... 2023-07-29T17:01:59.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-29T17:02:00.249 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[23526]: cluster 2023-07-29T17:01:59.082735+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:00.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:00 smithi078 bash[23704]: cluster 2023-07-29T17:01:59.082735+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:00.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:00 smithi078 bash[28259]: cluster 2023-07-29T17:01:59.082735+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:00.863 INFO:journalctl@ceph.osd.5.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:02:00.864 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-29T17:02:00.864 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2023-07-29T17:02:00.864 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2023-07-29T17:02:00.864 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2023-07-29T17:02:00.864 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-29T17:02:00.865 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-29T17:02:00.865 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:00 smithi129 bash[29247]: --> ceph-volume lvm activate successful for osd ID: 5 2023-07-29T17:02:02.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:02 smithi129 bash[23526]: cluster 2023-07-29T17:02:01.083293+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:02.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:02 smithi078 bash[23704]: cluster 2023-07-29T17:02:01.083293+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:02.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:02 smithi078 bash[28259]: cluster 2023-07-29T17:02:01.083293+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:03.659 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:03 smithi129 bash[29407]: debug 2023-07-29T17:02:03.492+0000 7f2ddf5af700 1 -- 172.21.15.129:0/1906761372 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f971c84e0 con 0x564f97241400 2023-07-29T17:02:04.151 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:03 smithi129 bash[29407]: debug 2023-07-29T17:02:03.800+0000 7f2de703a080 -1 Falling back to public interface 2023-07-29T17:02:04.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:04 smithi129 bash[23526]: cluster 2023-07-29T17:02:03.083869+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:04.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:04 smithi129 bash[23526]: audit 2023-07-29T17:02:03.653991+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:04.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:04 smithi078 bash[23704]: cluster 2023-07-29T17:02:03.083869+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:04.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:04 smithi078 bash[23704]: audit 2023-07-29T17:02:03.653991+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:04.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:04 smithi078 bash[28259]: cluster 2023-07-29T17:02:03.083869+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:04.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:04 smithi078 bash[28259]: audit 2023-07-29T17:02:03.653991+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:05.808 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-29T17:02:05.809 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-29T17:02:05.809 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2023-07-29T17:02:05.809 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2023-07-29T17:02:05.809 INFO:teuthology.orchestra.run.smithi129.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0240321 s, 436 MB/s 2023-07-29T17:02:05.809 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2023-07-29T17:02:06.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:06 smithi129 bash[23526]: cluster 2023-07-29T17:02:05.084435+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:06.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:06 smithi078 bash[23704]: cluster 2023-07-29T17:02:05.084435+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:06.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:06 smithi078 bash[28259]: cluster 2023-07-29T17:02:05.084435+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:07.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2023-07-29T17:02:07.157 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:06 smithi129 bash[29407]: debug 2023-07-29T17:02:06.804+0000 7f2de703a080 -1 osd.5 0 log_to_monitors {default=true} 2023-07-29T17:02:07.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:07 smithi129 bash[23526]: audit 2023-07-29T17:02:06.815416+0000 mon.b (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:02:07.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:07 smithi129 bash[23526]: audit 2023-07-29T17:02:06.816773+0000 mon.a (mon.0) 386 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:02:07.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:07 smithi078 bash[23704]: audit 2023-07-29T17:02:06.815416+0000 mon.b (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:02:07.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:07 smithi078 bash[23704]: audit 2023-07-29T17:02:06.816773+0000 mon.a (mon.0) 386 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:02:07.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:07 smithi078 bash[28259]: audit 2023-07-29T17:02:06.815416+0000 mon.b (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:02:07.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:07 smithi078 bash[28259]: audit 2023-07-29T17:02:06.816773+0000 mon.a (mon.0) 386 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:02:08.356 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[29407]: debug 2023-07-29T17:02:08.172+0000 7f2ddd5ab700 -1 osd.5 0 waiting for initial osdmap 2023-07-29T17:02:08.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[23526]: cluster 2023-07-29T17:02:07.085058+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:08.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[23526]: audit 2023-07-29T17:02:07.165134+0000 mon.a (mon.0) 387 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T17:02:08.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[23526]: cluster 2023-07-29T17:02:07.165241+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-07-29T17:02:08.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[23526]: audit 2023-07-29T17:02:07.165487+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:08.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[23526]: audit 2023-07-29T17:02:07.166910+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:08.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:08 smithi129 bash[23526]: audit 2023-07-29T17:02:07.168191+0000 mon.a (mon.0) 390 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:08.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[23704]: cluster 2023-07-29T17:02:07.085058+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:08.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[23704]: audit 2023-07-29T17:02:07.165134+0000 mon.a (mon.0) 387 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T17:02:08.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[23704]: cluster 2023-07-29T17:02:07.165241+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-07-29T17:02:08.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[23704]: audit 2023-07-29T17:02:07.165487+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:08.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[23704]: audit 2023-07-29T17:02:07.166910+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:08.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[23704]: audit 2023-07-29T17:02:07.168191+0000 mon.a (mon.0) 390 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:08.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[28259]: cluster 2023-07-29T17:02:07.085058+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:08.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[28259]: audit 2023-07-29T17:02:07.165134+0000 mon.a (mon.0) 387 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T17:02:08.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[28259]: cluster 2023-07-29T17:02:07.165241+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-07-29T17:02:08.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[28259]: audit 2023-07-29T17:02:07.165487+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:08.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[28259]: audit 2023-07-29T17:02:07.166910+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:08.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:08 smithi078 bash[28259]: audit 2023-07-29T17:02:07.168191+0000 mon.a (mon.0) 390 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:09.475 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:09 smithi129 bash[23526]: audit 2023-07-29T17:02:08.170729+0000 mon.a (mon.0) 391 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:09.476 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:09 smithi129 bash[23526]: cluster 2023-07-29T17:02:08.171008+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-07-29T17:02:09.476 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:09 smithi129 bash[23526]: audit 2023-07-29T17:02:08.172076+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:09.476 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:09 smithi129 bash[23526]: audit 2023-07-29T17:02:08.178425+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:09.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[23704]: audit 2023-07-29T17:02:08.170729+0000 mon.a (mon.0) 391 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:09.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[23704]: cluster 2023-07-29T17:02:08.171008+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-07-29T17:02:09.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[23704]: audit 2023-07-29T17:02:08.172076+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:09.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[23704]: audit 2023-07-29T17:02:08.178425+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:09.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[28259]: audit 2023-07-29T17:02:08.170729+0000 mon.a (mon.0) 391 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:09.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[28259]: cluster 2023-07-29T17:02:08.171008+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-07-29T17:02:09.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[28259]: audit 2023-07-29T17:02:08.172076+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:09.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:09 smithi078 bash[28259]: audit 2023-07-29T17:02:08.178425+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:10.492 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: cluster 2023-07-29T17:02:07.858975+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:02:10.493 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: cluster 2023-07-29T17:02:07.859247+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:02:10.493 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: cluster 2023-07-29T17:02:09.085639+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:10.493 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: cluster 2023-07-29T17:02:09.172556+0000 mon.a (mon.0) 395 : cluster [INF] osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734] boot 2023-07-29T17:02:10.493 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: cluster 2023-07-29T17:02:09.172665+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2023-07-29T17:02:10.493 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: audit 2023-07-29T17:02:09.172980+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:10.494 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:10 smithi129 bash[23526]: cluster 2023-07-29T17:02:10.177600+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-07-29T17:02:10.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: cluster 2023-07-29T17:02:07.858975+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:02:10.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: cluster 2023-07-29T17:02:07.859247+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:02:10.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: cluster 2023-07-29T17:02:09.085639+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:10.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: cluster 2023-07-29T17:02:09.172556+0000 mon.a (mon.0) 395 : cluster [INF] osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734] boot 2023-07-29T17:02:10.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: cluster 2023-07-29T17:02:09.172665+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2023-07-29T17:02:10.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: audit 2023-07-29T17:02:09.172980+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:10.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[23704]: cluster 2023-07-29T17:02:10.177600+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-07-29T17:02:10.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: cluster 2023-07-29T17:02:07.858975+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:02:10.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: cluster 2023-07-29T17:02:07.859247+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:02:10.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: cluster 2023-07-29T17:02:09.085639+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-29T17:02:10.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: cluster 2023-07-29T17:02:09.172556+0000 mon.a (mon.0) 395 : cluster [INF] osd.5 [v2:172.21.15.129:6808/2755329734,v1:172.21.15.129:6809/2755329734] boot 2023-07-29T17:02:10.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: cluster 2023-07-29T17:02:09.172665+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2023-07-29T17:02:10.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: audit 2023-07-29T17:02:09.172980+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:02:10.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:10 smithi078 bash[28259]: cluster 2023-07-29T17:02:10.177600+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-07-29T17:02:11.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: cephadm 2023-07-29T17:02:10.647630+0000 mgr.y (mgr.14144) 144 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:11.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.657450+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:11.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.665222+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:11.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.667395+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:11.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.674063+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:11.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.887253+0000 mgr.y (mgr.14144) 145 : audit [DBG] from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:02:11.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.888933+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:02:11.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: cephadm 2023-07-29T17:02:10.892002+0000 mgr.y (mgr.14144) 146 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:02:11.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.892510+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:02:11.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: audit 2023-07-29T17:02:10.894228+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:11.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: cluster 2023-07-29T17:02:11.086137+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:11.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[23704]: cluster 2023-07-29T17:02:11.176310+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-07-29T17:02:11.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: cephadm 2023-07-29T17:02:10.647630+0000 mgr.y (mgr.14144) 144 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:11.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.657450+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:11.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.665222+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:11.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.667395+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:11.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.674063+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:11.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.887253+0000 mgr.y (mgr.14144) 145 : audit [DBG] from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:02:11.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.888933+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:02:11.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: cephadm 2023-07-29T17:02:10.892002+0000 mgr.y (mgr.14144) 146 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:02:11.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.892510+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:02:11.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: audit 2023-07-29T17:02:10.894228+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:11.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: cluster 2023-07-29T17:02:11.086137+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:11.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:11 smithi078 bash[28259]: cluster 2023-07-29T17:02:11.176310+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-07-29T17:02:12.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: cephadm 2023-07-29T17:02:10.647630+0000 mgr.y (mgr.14144) 144 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:12.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.657450+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:12.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.665222+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:12.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.667395+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:12.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.674063+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:12.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.887253+0000 mgr.y (mgr.14144) 145 : audit [DBG] from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:02:12.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.888933+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:02:12.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: cephadm 2023-07-29T17:02:10.892002+0000 mgr.y (mgr.14144) 146 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:02:12.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.892510+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:02:12.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: audit 2023-07-29T17:02:10.894228+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:12.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: cluster 2023-07-29T17:02:11.086137+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:12.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:11 smithi129 bash[23526]: cluster 2023-07-29T17:02:11.176310+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-07-29T17:02:14.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:14 smithi129 bash[23526]: cluster 2023-07-29T17:02:13.086708+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:14.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:14 smithi078 bash[28259]: cluster 2023-07-29T17:02:13.086708+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:14.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:14 smithi078 bash[23704]: cluster 2023-07-29T17:02:13.086708+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:15.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:15 smithi129 bash[23526]: audit 2023-07-29T17:02:14.535825+0000 mon.b (mon.1) 16 : audit [INF] from='client.? 172.21.15.129:0/3131579245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]: dispatch 2023-07-29T17:02:15.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:15 smithi129 bash[23526]: audit 2023-07-29T17:02:14.537185+0000 mon.a (mon.0) 407 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]: dispatch 2023-07-29T17:02:15.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:15 smithi129 bash[23526]: audit 2023-07-29T17:02:14.546105+0000 mon.a (mon.0) 408 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]': finished 2023-07-29T17:02:15.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:15 smithi129 bash[23526]: cluster 2023-07-29T17:02:14.546288+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2023-07-29T17:02:15.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:15 smithi129 bash[23526]: audit 2023-07-29T17:02:14.546552+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:15.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[23704]: audit 2023-07-29T17:02:14.535825+0000 mon.b (mon.1) 16 : audit [INF] from='client.? 172.21.15.129:0/3131579245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]: dispatch 2023-07-29T17:02:15.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[23704]: audit 2023-07-29T17:02:14.537185+0000 mon.a (mon.0) 407 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]: dispatch 2023-07-29T17:02:15.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[23704]: audit 2023-07-29T17:02:14.546105+0000 mon.a (mon.0) 408 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]': finished 2023-07-29T17:02:15.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[23704]: cluster 2023-07-29T17:02:14.546288+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2023-07-29T17:02:15.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[23704]: audit 2023-07-29T17:02:14.546552+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:15.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[28259]: audit 2023-07-29T17:02:14.535825+0000 mon.b (mon.1) 16 : audit [INF] from='client.? 172.21.15.129:0/3131579245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]: dispatch 2023-07-29T17:02:15.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[28259]: audit 2023-07-29T17:02:14.537185+0000 mon.a (mon.0) 407 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]: dispatch 2023-07-29T17:02:15.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[28259]: audit 2023-07-29T17:02:14.546105+0000 mon.a (mon.0) 408 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "392bb467-d2ee-441f-86f0-79e664b204ba"}]': finished 2023-07-29T17:02:15.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[28259]: cluster 2023-07-29T17:02:14.546288+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2023-07-29T17:02:15.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:15 smithi078 bash[28259]: audit 2023-07-29T17:02:14.546552+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:16.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:16 smithi129 bash[23526]: cluster 2023-07-29T17:02:15.087277+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:16.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:16 smithi129 bash[23526]: audit 2023-07-29T17:02:15.206531+0000 mon.b (mon.1) 17 : audit [DBG] from='client.? 172.21.15.129:0/36187813' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:02:16.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:16 smithi078 bash[28259]: cluster 2023-07-29T17:02:15.087277+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:16.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:16 smithi078 bash[28259]: audit 2023-07-29T17:02:15.206531+0000 mon.b (mon.1) 17 : audit [DBG] from='client.? 172.21.15.129:0/36187813' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:02:16.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:16 smithi078 bash[23704]: cluster 2023-07-29T17:02:15.087277+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:16.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:16 smithi078 bash[23704]: audit 2023-07-29T17:02:15.206531+0000 mon.b (mon.1) 17 : audit [DBG] from='client.? 172.21.15.129:0/36187813' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:02:18.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:18 smithi129 bash[23526]: cluster 2023-07-29T17:02:17.088011+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:18.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:18 smithi078 bash[28259]: cluster 2023-07-29T17:02:17.088011+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:18.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:18 smithi078 bash[23704]: cluster 2023-07-29T17:02:17.088011+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:20.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:20 smithi078 bash[23704]: cluster 2023-07-29T17:02:19.088601+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:20.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:20 smithi078 bash[28259]: cluster 2023-07-29T17:02:19.088601+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:20.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:20 smithi129 bash[23526]: cluster 2023-07-29T17:02:19.088601+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:22.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:22 smithi078 bash[23704]: cluster 2023-07-29T17:02:21.089268+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:22.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:22 smithi078 bash[28259]: cluster 2023-07-29T17:02:21.089268+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:22.617 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:22 smithi129 bash[23526]: cluster 2023-07-29T17:02:21.089268+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:23.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:23 smithi078 bash[23704]: audit 2023-07-29T17:02:22.374164+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:02:23.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:23 smithi078 bash[23704]: audit 2023-07-29T17:02:22.376188+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:23.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:23 smithi078 bash[28259]: audit 2023-07-29T17:02:22.374164+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:02:23.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:23 smithi078 bash[28259]: audit 2023-07-29T17:02:22.376188+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:23.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:23 smithi129 bash[23526]: audit 2023-07-29T17:02:22.374164+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:02:23.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:23 smithi129 bash[23526]: audit 2023-07-29T17:02:22.376188+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:24.490 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:24 smithi129 bash[23526]: cephadm 2023-07-29T17:02:22.377892+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon osd.6 on smithi129 2023-07-29T17:02:24.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:24 smithi129 bash[23526]: cluster 2023-07-29T17:02:23.089974+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:24.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:24 smithi078 bash[28259]: cephadm 2023-07-29T17:02:22.377892+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon osd.6 on smithi129 2023-07-29T17:02:24.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:24 smithi078 bash[28259]: cluster 2023-07-29T17:02:23.089974+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:24.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:24 smithi078 bash[23704]: cephadm 2023-07-29T17:02:22.377892+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon osd.6 on smithi129 2023-07-29T17:02:24.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:24 smithi078 bash[23704]: cluster 2023-07-29T17:02:23.089974+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:24.753 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 6 on host 'smithi129' 2023-07-29T17:02:25.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:25 smithi129 bash[23526]: audit 2023-07-29T17:02:24.727462+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:25.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:25 smithi129 bash[23526]: audit 2023-07-29T17:02:24.735971+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:25.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:25 smithi129 bash[23526]: audit 2023-07-29T17:02:24.747079+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:25.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:25 smithi078 bash[23704]: audit 2023-07-29T17:02:24.727462+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:25.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:25 smithi078 bash[23704]: audit 2023-07-29T17:02:24.735971+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:25.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:25 smithi078 bash[23704]: audit 2023-07-29T17:02:24.747079+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:25.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:25 smithi078 bash[28259]: audit 2023-07-29T17:02:24.727462+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:25.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:25 smithi078 bash[28259]: audit 2023-07-29T17:02:24.735971+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:25.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:25 smithi078 bash[28259]: audit 2023-07-29T17:02:24.747079+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:25.926 DEBUG:teuthology.orchestra.run.smithi129:osd.6> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.6.service 2023-07-29T17:02:25.929 INFO:tasks.cephadm:Deploying osd.7 on smithi129 with /dev/vg_nvme/lv_1... 2023-07-29T17:02:25.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-29T17:02:26.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:26 smithi129 bash[23526]: cluster 2023-07-29T17:02:25.090579+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:26.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:26 smithi078 bash[28259]: cluster 2023-07-29T17:02:25.090579+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:26.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:26 smithi078 bash[23704]: cluster 2023-07-29T17:02:25.090579+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:26.770 INFO:journalctl@ceph.osd.6.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:02:27.409 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-29T17:02:27.410 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-6 --no-mon-config 2023-07-29T17:02:27.410 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6/block 2023-07-29T17:02:27.410 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block 2023-07-29T17:02:27.410 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-29T17:02:27.410 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-29T17:02:27.410 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:27 smithi129 bash[31988]: --> ceph-volume lvm activate successful for osd ID: 6 2023-07-29T17:02:28.283 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:28 smithi129 bash[23526]: cluster 2023-07-29T17:02:27.091199+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:28.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:28 smithi078 bash[23704]: cluster 2023-07-29T17:02:27.091199+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:28.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:28 smithi078 bash[28259]: cluster 2023-07-29T17:02:27.091199+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:30.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:29 smithi129 bash[23526]: audit 2023-07-29T17:02:28.933514+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:30.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:29 smithi129 bash[23526]: cluster 2023-07-29T17:02:29.091791+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:30.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:29 smithi078 bash[23704]: audit 2023-07-29T17:02:28.933514+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:30.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:29 smithi078 bash[23704]: cluster 2023-07-29T17:02:29.091791+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:30.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:29 smithi078 bash[28259]: audit 2023-07-29T17:02:28.933514+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:30.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:29 smithi078 bash[28259]: cluster 2023-07-29T17:02:29.091791+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:30.780 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:30 smithi129 bash[32184]: debug 2023-07-29T17:02:30.511+0000 7f77ab743700 1 -- 172.21.15.129:0/2327413818 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff158a24e0 con 0x55ff1591ac00 2023-07-29T17:02:31.409 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:31 smithi129 bash[32184]: debug 2023-07-29T17:02:31.071+0000 7f77b31ce080 -1 Falling back to public interface 2023-07-29T17:02:31.676 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-29T17:02:31.676 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-29T17:02:31.677 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2023-07-29T17:02:31.677 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2023-07-29T17:02:31.677 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0142003 s, 738 MB/s 2023-07-29T17:02:31.677 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2023-07-29T17:02:32.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:32 smithi129 bash[23526]: cluster 2023-07-29T17:02:31.092403+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:32.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:32 smithi078 bash[23704]: cluster 2023-07-29T17:02:31.092403+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:32.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:32 smithi078 bash[28259]: cluster 2023-07-29T17:02:31.092403+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:32.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2023-07-29T17:02:34.255 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:34 smithi129 bash[23526]: cluster 2023-07-29T17:02:33.092947+0000 mgr.y (mgr.14144) 159 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:34.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:34 smithi129 bash[23526]: audit 2023-07-29T17:02:34.075525+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:02:34.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:34 smithi129 bash[23526]: audit 2023-07-29T17:02:34.076852+0000 mon.a (mon.0) 417 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:02:34.256 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:34 smithi129 bash[32184]: debug 2023-07-29T17:02:34.063+0000 7f77b31ce080 -1 osd.6 0 log_to_monitors {default=true} 2023-07-29T17:02:34.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:34 smithi078 bash[23704]: cluster 2023-07-29T17:02:33.092947+0000 mgr.y (mgr.14144) 159 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:34.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:34 smithi078 bash[23704]: audit 2023-07-29T17:02:34.075525+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:02:34.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:34 smithi078 bash[23704]: audit 2023-07-29T17:02:34.076852+0000 mon.a (mon.0) 417 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:02:34.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:34 smithi078 bash[28259]: cluster 2023-07-29T17:02:33.092947+0000 mgr.y (mgr.14144) 159 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:34.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:34 smithi078 bash[28259]: audit 2023-07-29T17:02:34.075525+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:02:34.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:34 smithi078 bash[28259]: audit 2023-07-29T17:02:34.076852+0000 mon.a (mon.0) 417 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:02:35.659 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:02:35 smithi129 bash[32184]: debug 2023-07-29T17:02:35.179+0000 7f77a973f700 -1 osd.6 0 waiting for initial osdmap 2023-07-29T17:02:35.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:35 smithi129 bash[23526]: audit 2023-07-29T17:02:34.168323+0000 mon.a (mon.0) 418 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T17:02:35.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:35 smithi129 bash[23526]: cluster 2023-07-29T17:02:34.168484+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2023-07-29T17:02:35.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:35 smithi129 bash[23526]: audit 2023-07-29T17:02:34.168757+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:35.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:35 smithi129 bash[23526]: audit 2023-07-29T17:02:34.169788+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:35.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:35 smithi129 bash[23526]: audit 2023-07-29T17:02:34.171007+0000 mon.a (mon.0) 421 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:35.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[28259]: audit 2023-07-29T17:02:34.168323+0000 mon.a (mon.0) 418 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T17:02:35.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[28259]: cluster 2023-07-29T17:02:34.168484+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2023-07-29T17:02:35.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[28259]: audit 2023-07-29T17:02:34.168757+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:35.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[28259]: audit 2023-07-29T17:02:34.169788+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:35.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[28259]: audit 2023-07-29T17:02:34.171007+0000 mon.a (mon.0) 421 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:35.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[23704]: audit 2023-07-29T17:02:34.168323+0000 mon.a (mon.0) 418 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T17:02:35.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[23704]: cluster 2023-07-29T17:02:34.168484+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2023-07-29T17:02:35.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[23704]: audit 2023-07-29T17:02:34.168757+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:35.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[23704]: audit 2023-07-29T17:02:34.169788+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:35.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:35 smithi078 bash[23704]: audit 2023-07-29T17:02:34.171007+0000 mon.a (mon.0) 421 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:36.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: cluster 2023-07-29T17:02:35.093515+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:35.175961+0000 mon.a (mon.0) 422 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: cluster 2023-07-29T17:02:35.176136+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2023-07-29T17:02:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:35.177151+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:35.186063+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:35.245699+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:02:36.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:35.248442+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:35.897228+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: cluster 2023-07-29T17:02:36.112501+0000 mon.a (mon.0) 429 : cluster [INF] osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648] boot 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: cluster 2023-07-29T17:02:36.112633+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:36.113347+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:36.151346+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:36.155563+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:36.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:36 smithi129 bash[23526]: audit 2023-07-29T17:02:36.163564+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: cluster 2023-07-29T17:02:35.093515+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:35.175961+0000 mon.a (mon.0) 422 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: cluster 2023-07-29T17:02:35.176136+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2023-07-29T17:02:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:35.177151+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:35.186063+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:35.245699+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:02:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:35.248442+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:35.897228+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: cluster 2023-07-29T17:02:36.112501+0000 mon.a (mon.0) 429 : cluster [INF] osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648] boot 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: cluster 2023-07-29T17:02:36.112633+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:36.113347+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:36.151346+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:36.155563+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:36.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[23704]: audit 2023-07-29T17:02:36.163564+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: cluster 2023-07-29T17:02:35.093515+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:35.175961+0000 mon.a (mon.0) 422 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: cluster 2023-07-29T17:02:35.176136+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:35.177151+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:35.186063+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:35.245699+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:02:36.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:35.248442+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:35.897228+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: cluster 2023-07-29T17:02:36.112501+0000 mon.a (mon.0) 429 : cluster [INF] osd.6 [v2:172.21.15.129:6816/3214745648,v1:172.21.15.129:6817/3214745648] boot 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: cluster 2023-07-29T17:02:36.112633+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:36.113347+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:36.151346+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:36.155563+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:36.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:36 smithi078 bash[28259]: audit 2023-07-29T17:02:36.163564+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:37.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: cluster 2023-07-29T17:02:35.096346+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:02:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: cluster 2023-07-29T17:02:35.096541+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:02:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: cephadm 2023-07-29T17:02:35.887474+0000 mgr.y (mgr.14144) 161 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: audit 2023-07-29T17:02:36.469492+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:02:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: audit 2023-07-29T17:02:36.473261+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:02:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: audit 2023-07-29T17:02:36.475070+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:37.606 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:37 smithi129 bash[23526]: cluster 2023-07-29T17:02:37.112508+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2023-07-29T17:02:37.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: cluster 2023-07-29T17:02:35.096346+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:02:37.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: cluster 2023-07-29T17:02:35.096541+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:02:37.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: cephadm 2023-07-29T17:02:35.887474+0000 mgr.y (mgr.14144) 161 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:37.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: audit 2023-07-29T17:02:36.469492+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:02:37.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: audit 2023-07-29T17:02:36.473261+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:02:37.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: audit 2023-07-29T17:02:36.475070+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:37.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[23704]: cluster 2023-07-29T17:02:37.112508+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2023-07-29T17:02:37.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: cluster 2023-07-29T17:02:35.096346+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:02:37.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: cluster 2023-07-29T17:02:35.096541+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:02:37.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: cephadm 2023-07-29T17:02:35.887474+0000 mgr.y (mgr.14144) 161 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:37.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: audit 2023-07-29T17:02:36.469492+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T17:02:37.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: audit 2023-07-29T17:02:36.473261+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T17:02:37.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: audit 2023-07-29T17:02:36.475070+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:37.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:37 smithi078 bash[28259]: cluster 2023-07-29T17:02:37.112508+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2023-07-29T17:02:38.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:38 smithi129 bash[23526]: audit 2023-07-29T17:02:36.467789+0000 mgr.y (mgr.14144) 162 : audit [DBG] from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:02:38.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:38 smithi129 bash[23526]: cephadm 2023-07-29T17:02:36.472681+0000 mgr.y (mgr.14144) 163 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:02:38.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:38 smithi129 bash[23526]: cluster 2023-07-29T17:02:37.094110+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v154: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:38.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:38 smithi129 bash[23526]: cluster 2023-07-29T17:02:38.114905+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2023-07-29T17:02:38.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[28259]: audit 2023-07-29T17:02:36.467789+0000 mgr.y (mgr.14144) 162 : audit [DBG] from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:02:38.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[28259]: cephadm 2023-07-29T17:02:36.472681+0000 mgr.y (mgr.14144) 163 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:02:38.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[28259]: cluster 2023-07-29T17:02:37.094110+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v154: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:38.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[28259]: cluster 2023-07-29T17:02:38.114905+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2023-07-29T17:02:38.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[23704]: audit 2023-07-29T17:02:36.467789+0000 mgr.y (mgr.14144) 162 : audit [DBG] from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:02:38.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[23704]: cephadm 2023-07-29T17:02:36.472681+0000 mgr.y (mgr.14144) 163 : cephadm [INF] Applying service osd.None on host smithi129... 2023-07-29T17:02:38.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[23704]: cluster 2023-07-29T17:02:37.094110+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v154: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:38.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:38 smithi078 bash[23704]: cluster 2023-07-29T17:02:38.114905+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2023-07-29T17:02:39.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:39 smithi129 bash[23526]: cluster 2023-07-29T17:02:39.094678+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v157: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:39.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:39 smithi078 bash[28259]: cluster 2023-07-29T17:02:39.094678+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v157: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:39.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:39 smithi078 bash[23704]: cluster 2023-07-29T17:02:39.094678+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v157: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:40.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:40 smithi129 bash[23526]: audit 2023-07-29T17:02:40.077190+0000 mon.b (mon.1) 20 : audit [INF] from='client.? 172.21.15.129:0/1110446969' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]: dispatch 2023-07-29T17:02:40.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:40 smithi129 bash[23526]: audit 2023-07-29T17:02:40.078434+0000 mon.a (mon.0) 440 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]: dispatch 2023-07-29T17:02:40.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:40 smithi129 bash[23526]: audit 2023-07-29T17:02:40.087851+0000 mon.a (mon.0) 441 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]': finished 2023-07-29T17:02:40.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:40 smithi129 bash[23526]: cluster 2023-07-29T17:02:40.088074+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2023-07-29T17:02:40.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:40 smithi129 bash[23526]: audit 2023-07-29T17:02:40.088368+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:40.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[23704]: audit 2023-07-29T17:02:40.077190+0000 mon.b (mon.1) 20 : audit [INF] from='client.? 172.21.15.129:0/1110446969' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]: dispatch 2023-07-29T17:02:40.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[23704]: audit 2023-07-29T17:02:40.078434+0000 mon.a (mon.0) 440 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]: dispatch 2023-07-29T17:02:40.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[23704]: audit 2023-07-29T17:02:40.087851+0000 mon.a (mon.0) 441 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]': finished 2023-07-29T17:02:40.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[23704]: cluster 2023-07-29T17:02:40.088074+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2023-07-29T17:02:40.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[23704]: audit 2023-07-29T17:02:40.088368+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:40.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[28259]: audit 2023-07-29T17:02:40.077190+0000 mon.b (mon.1) 20 : audit [INF] from='client.? 172.21.15.129:0/1110446969' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]: dispatch 2023-07-29T17:02:40.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[28259]: audit 2023-07-29T17:02:40.078434+0000 mon.a (mon.0) 440 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]: dispatch 2023-07-29T17:02:40.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[28259]: audit 2023-07-29T17:02:40.087851+0000 mon.a (mon.0) 441 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "690a3c3e-18d4-493f-a3fa-176f897a16d1"}]': finished 2023-07-29T17:02:40.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[28259]: cluster 2023-07-29T17:02:40.088074+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2023-07-29T17:02:40.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:40 smithi078 bash[28259]: audit 2023-07-29T17:02:40.088368+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:41.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:41 smithi129 bash[23526]: audit 2023-07-29T17:02:40.736638+0000 mon.b (mon.1) 21 : audit [DBG] from='client.? 172.21.15.129:0/2361960701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:02:41.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:41 smithi129 bash[23526]: cluster 2023-07-29T17:02:41.095261+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v159: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:41.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:41 smithi078 bash[23704]: audit 2023-07-29T17:02:40.736638+0000 mon.b (mon.1) 21 : audit [DBG] from='client.? 172.21.15.129:0/2361960701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:02:41.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:41 smithi078 bash[23704]: cluster 2023-07-29T17:02:41.095261+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v159: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:41 smithi078 bash[28259]: audit 2023-07-29T17:02:40.736638+0000 mon.b (mon.1) 21 : audit [DBG] from='client.? 172.21.15.129:0/2361960701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T17:02:41.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:41 smithi078 bash[28259]: cluster 2023-07-29T17:02:41.095261+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v159: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:44.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:44 smithi129 bash[23526]: cluster 2023-07-29T17:02:43.095871+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:44.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:44 smithi078 bash[23704]: cluster 2023-07-29T17:02:43.095871+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:44.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:44 smithi078 bash[28259]: cluster 2023-07-29T17:02:43.095871+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:46.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:46 smithi078 bash[23704]: cluster 2023-07-29T17:02:45.096435+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:46.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:46 smithi078 bash[28259]: cluster 2023-07-29T17:02:45.096435+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:46.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:46 smithi129 bash[23526]: cluster 2023-07-29T17:02:45.096435+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:48.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:48 smithi129 bash[23526]: cluster 2023-07-29T17:02:47.097062+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:48.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:48 smithi129 bash[23526]: audit 2023-07-29T17:02:47.773354+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:02:48.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:48 smithi129 bash[23526]: audit 2023-07-29T17:02:47.775247+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:48.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:48 smithi078 bash[23704]: cluster 2023-07-29T17:02:47.097062+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:48.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:48 smithi078 bash[23704]: audit 2023-07-29T17:02:47.773354+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:02:48.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:48 smithi078 bash[23704]: audit 2023-07-29T17:02:47.775247+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:48.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:48 smithi078 bash[28259]: cluster 2023-07-29T17:02:47.097062+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:48.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:48 smithi078 bash[28259]: audit 2023-07-29T17:02:47.773354+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:02:48.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:48 smithi078 bash[28259]: audit 2023-07-29T17:02:47.775247+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:02:49.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:49 smithi129 bash[23526]: cephadm 2023-07-29T17:02:47.777064+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon osd.7 on smithi129 2023-07-29T17:02:49.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:49 smithi078 bash[23704]: cephadm 2023-07-29T17:02:47.777064+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon osd.7 on smithi129 2023-07-29T17:02:49.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:49 smithi078 bash[28259]: cephadm 2023-07-29T17:02:47.777064+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon osd.7 on smithi129 2023-07-29T17:02:50.047 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 7 on host 'smithi129' 2023-07-29T17:02:50.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:50 smithi129 bash[23526]: cluster 2023-07-29T17:02:49.097619+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:50.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:50 smithi129 bash[23526]: audit 2023-07-29T17:02:50.022331+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:50.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:50 smithi129 bash[23526]: audit 2023-07-29T17:02:50.032111+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:50.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:50 smithi129 bash[23526]: audit 2023-07-29T17:02:50.042023+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:50.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[23704]: cluster 2023-07-29T17:02:49.097619+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:50.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[23704]: audit 2023-07-29T17:02:50.022331+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:50.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[23704]: audit 2023-07-29T17:02:50.032111+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:50.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[23704]: audit 2023-07-29T17:02:50.042023+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:50.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[28259]: cluster 2023-07-29T17:02:49.097619+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:50.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[28259]: audit 2023-07-29T17:02:50.022331+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:50.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[28259]: audit 2023-07-29T17:02:50.032111+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:50.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:50 smithi078 bash[28259]: audit 2023-07-29T17:02:50.042023+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:51.216 DEBUG:teuthology.orchestra.run.smithi129:osd.7> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.7.service 2023-07-29T17:02:51.220 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-07-29T17:02:51.220 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd stat -f json 2023-07-29T17:02:51.455 INFO:journalctl@ceph.osd.7.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:02:52.159 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-29T17:02:52.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2023-07-29T17:02:52.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2023-07-29T17:02:52.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2023-07-29T17:02:52.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-29T17:02:52.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-29T17:02:52.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:51 smithi129 bash[34761]: --> ceph-volume lvm activate successful for osd ID: 7 2023-07-29T17:02:52.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:52 smithi129 bash[23526]: cluster 2023-07-29T17:02:51.098240+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:52.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:52 smithi078 bash[28259]: cluster 2023-07-29T17:02:51.098240+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:52.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:52 smithi078 bash[23704]: cluster 2023-07-29T17:02:51.098240+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:53.812 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:53 smithi129 bash[34912]: debug 2023-07-29T17:02:53.531+0000 7f46f9ad3700 1 -- 172.21.15.129:0/1872385975 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594102b24e0 con 0x55941032b000 2023-07-29T17:02:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:02:54.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:54 smithi129 bash[23526]: cluster 2023-07-29T17:02:53.098851+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:54.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:54 smithi129 bash[23526]: audit 2023-07-29T17:02:53.619649+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:54.410 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:54 smithi129 bash[34912]: debug 2023-07-29T17:02:54.075+0000 7f470155e080 -1 Falling back to public interface 2023-07-29T17:02:54.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:54 smithi078 bash[23704]: cluster 2023-07-29T17:02:53.098851+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:54.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:54 smithi078 bash[23704]: audit 2023-07-29T17:02:53.619649+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:54.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:54 smithi078 bash[28259]: cluster 2023-07-29T17:02:53.098851+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:54.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:54 smithi078 bash[28259]: audit 2023-07-29T17:02:53.619649+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:55.033 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":46,"num_osds":8,"num_up_osds":7,"osd_up_since":1690650156,"num_in_osds":8,"osd_in_since":1690650160,"num_remapped_pgs":0} 2023-07-29T17:02:55.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:55 smithi129 bash[23526]: audit 2023-07-29T17:02:54.309670+0000 mon.a (mon.0) 450 : audit [DBG] from='client.? 172.21.15.78:0/1541098056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:02:55.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:55 smithi078 bash[23704]: audit 2023-07-29T17:02:54.309670+0000 mon.a (mon.0) 450 : audit [DBG] from='client.? 172.21.15.78:0/1541098056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:02:55.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:55 smithi078 bash[28259]: audit 2023-07-29T17:02:54.309670+0000 mon.a (mon.0) 450 : audit [DBG] from='client.? 172.21.15.78:0/1541098056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:02:56.034 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd stat -f json 2023-07-29T17:02:56.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:56 smithi078 bash[23704]: cluster 2023-07-29T17:02:55.099463+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:56.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:56 smithi078 bash[28259]: cluster 2023-07-29T17:02:55.099463+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:56.478 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:56 smithi129 bash[23526]: cluster 2023-07-29T17:02:55.099463+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:57.369 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:57 smithi078 bash[23704]: audit 2023-07-29T17:02:57.037387+0000 mon.b (mon.1) 22 : audit [INF] from='osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:02:57.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:57 smithi078 bash[23704]: audit 2023-07-29T17:02:57.038589+0000 mon.a (mon.0) 451 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:02:57.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:57 smithi078 bash[23704]: cluster 2023-07-29T17:02:57.100038+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:57.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:57 smithi078 bash[28259]: audit 2023-07-29T17:02:57.037387+0000 mon.b (mon.1) 22 : audit [INF] from='osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:02:57.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:57 smithi078 bash[28259]: audit 2023-07-29T17:02:57.038589+0000 mon.a (mon.0) 451 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:02:57.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:57 smithi078 bash[28259]: cluster 2023-07-29T17:02:57.100038+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:57.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:57 smithi129 bash[23526]: audit 2023-07-29T17:02:57.037387+0000 mon.b (mon.1) 22 : audit [INF] from='osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:02:57.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:57 smithi129 bash[23526]: audit 2023-07-29T17:02:57.038589+0000 mon.a (mon.0) 451 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:02:57.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:57 smithi129 bash[23526]: cluster 2023-07-29T17:02:57.100038+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:57.410 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:57 smithi129 bash[34912]: debug 2023-07-29T17:02:57.027+0000 7f470155e080 -1 osd.7 0 log_to_monitors {default=true} 2023-07-29T17:02:58.659 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:02:58 smithi129 bash[34912]: debug 2023-07-29T17:02:58.215+0000 7f46f7acf700 -1 osd.7 0 waiting for initial osdmap 2023-07-29T17:02:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:58 smithi129 bash[23526]: audit 2023-07-29T17:02:57.209156+0000 mon.a (mon.0) 452 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T17:02:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:58 smithi129 bash[23526]: cluster 2023-07-29T17:02:57.209280+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2023-07-29T17:02:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:58 smithi129 bash[23526]: audit 2023-07-29T17:02:57.209591+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:58 smithi129 bash[23526]: audit 2023-07-29T17:02:57.213187+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:58 smithi129 bash[23526]: audit 2023-07-29T17:02:57.214170+0000 mon.a (mon.0) 455 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:58.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[23704]: audit 2023-07-29T17:02:57.209156+0000 mon.a (mon.0) 452 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T17:02:58.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[23704]: cluster 2023-07-29T17:02:57.209280+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2023-07-29T17:02:58.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[23704]: audit 2023-07-29T17:02:57.209591+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:58.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[23704]: audit 2023-07-29T17:02:57.213187+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:58.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[23704]: audit 2023-07-29T17:02:57.214170+0000 mon.a (mon.0) 455 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:58.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[28259]: audit 2023-07-29T17:02:57.209156+0000 mon.a (mon.0) 452 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T17:02:58.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[28259]: cluster 2023-07-29T17:02:57.209280+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2023-07-29T17:02:58.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[28259]: audit 2023-07-29T17:02:57.209591+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:58.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[28259]: audit 2023-07-29T17:02:57.213187+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:58.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:58 smithi078 bash[28259]: audit 2023-07-29T17:02:57.214170+0000 mon.a (mon.0) 455 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:02:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:02:59.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.214130+0000 mon.a (mon.0) 456 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:59.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: cluster 2023-07-29T17:02:58.214553+0000 mon.a (mon.0) 457 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2023-07-29T17:02:59.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.215563+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.222078+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: cephadm 2023-07-29T17:02:58.609529+0000 mgr.y (mgr.14144) 176 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:59.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.620048+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.626967+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.629713+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:58.637360+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:59.071381+0000 mon.a (mon.0) 464 : audit [DBG] from='client.? 172.21.15.78:0/3060468588' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: cluster 2023-07-29T17:02:59.100571+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: cluster 2023-07-29T17:02:59.215562+0000 mon.a (mon.0) 465 : cluster [INF] osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209] boot 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: cluster 2023-07-29T17:02:59.215628+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-07-29T17:02:59.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[23704]: audit 2023-07-29T17:02:59.215926+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.214130+0000 mon.a (mon.0) 456 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:59.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: cluster 2023-07-29T17:02:58.214553+0000 mon.a (mon.0) 457 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2023-07-29T17:02:59.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.215563+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.222078+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: cephadm 2023-07-29T17:02:58.609529+0000 mgr.y (mgr.14144) 176 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.620048+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.626967+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.629713+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:58.637360+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:59.071381+0000 mon.a (mon.0) 464 : audit [DBG] from='client.? 172.21.15.78:0/3060468588' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: cluster 2023-07-29T17:02:59.100571+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: cluster 2023-07-29T17:02:59.215562+0000 mon.a (mon.0) 465 : cluster [INF] osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209] boot 2023-07-29T17:02:59.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: cluster 2023-07-29T17:02:59.215628+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-07-29T17:02:59.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:02:59 smithi078 bash[28259]: audit 2023-07-29T17:02:59.215926+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.214130+0000 mon.a (mon.0) 456 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2023-07-29T17:02:59.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: cluster 2023-07-29T17:02:58.214553+0000 mon.a (mon.0) 457 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2023-07-29T17:02:59.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.215563+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.222078+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: cephadm 2023-07-29T17:02:58.609529+0000 mgr.y (mgr.14144) 176 : cephadm [INF] Detected new or changed devices on smithi129 2023-07-29T17:02:59.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.620048+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.626967+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.629713+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:58.637360+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:59.071381+0000 mon.a (mon.0) 464 : audit [DBG] from='client.? 172.21.15.78:0/3060468588' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: cluster 2023-07-29T17:02:59.100571+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: cluster 2023-07-29T17:02:59.215562+0000 mon.a (mon.0) 465 : cluster [INF] osd.7 [v2:172.21.15.129:6824/815409209,v1:172.21.15.129:6825/815409209] boot 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: cluster 2023-07-29T17:02:59.215628+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-07-29T17:02:59.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:02:59 smithi129 bash[23526]: audit 2023-07-29T17:02:59.215926+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:02:59.738 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":48,"num_osds":8,"num_up_osds":7,"osd_up_since":1690650156,"num_in_osds":8,"osd_in_since":1690650160,"num_remapped_pgs":0} 2023-07-29T17:03:00.739 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd stat -f json 2023-07-29T17:03:00.909 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:00 smithi129 bash[23526]: cluster 2023-07-29T17:02:57.985852+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:03:00.909 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:00 smithi129 bash[23526]: cluster 2023-07-29T17:02:57.986048+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:03:00.910 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:00 smithi129 bash[23526]: cluster 2023-07-29T17:03:00.218860+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2023-07-29T17:03:00.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:00 smithi078 bash[28259]: cluster 2023-07-29T17:02:57.985852+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:03:00.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:00 smithi078 bash[28259]: cluster 2023-07-29T17:02:57.986048+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:03:00.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:00 smithi078 bash[28259]: cluster 2023-07-29T17:03:00.218860+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2023-07-29T17:03:00.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:00 smithi078 bash[23704]: cluster 2023-07-29T17:02:57.985852+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-29T17:03:00.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:00 smithi078 bash[23704]: cluster 2023-07-29T17:02:57.986048+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-29T17:03:00.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:00 smithi078 bash[23704]: cluster 2023-07-29T17:03:00.218860+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2023-07-29T17:03:02.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:02 smithi129 bash[23526]: cluster 2023-07-29T17:03:01.101094+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:02.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:02 smithi129 bash[23526]: cluster 2023-07-29T17:03:01.219057+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2023-07-29T17:03:02.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:02 smithi078 bash[28259]: cluster 2023-07-29T17:03:01.101094+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:02.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:02 smithi078 bash[28259]: cluster 2023-07-29T17:03:01.219057+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2023-07-29T17:03:02.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:02 smithi078 bash[23704]: cluster 2023-07-29T17:03:01.101094+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:02.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:02 smithi078 bash[23704]: cluster 2023-07-29T17:03:01.219057+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2023-07-29T17:03:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:03:04.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:04 smithi129 bash[23526]: cluster 2023-07-29T17:03:03.101697+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:04.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:04 smithi129 bash[23526]: audit 2023-07-29T17:03:03.816613+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.78:0/913524371' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:03:04.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:04 smithi078 bash[23704]: cluster 2023-07-29T17:03:03.101697+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:04.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:04 smithi078 bash[23704]: audit 2023-07-29T17:03:03.816613+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.78:0/913524371' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:03:04.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:04 smithi078 bash[28259]: cluster 2023-07-29T17:03:03.101697+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:04.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:04 smithi078 bash[28259]: audit 2023-07-29T17:03:03.816613+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.78:0/913524371' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T17:03:04.493 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1690650179,"num_in_osds":8,"osd_in_since":1690650160,"num_remapped_pgs":0} 2023-07-29T17:03:04.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd dump --format=json 2023-07-29T17:03:06.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:06 smithi129 bash[23526]: cluster 2023-07-29T17:03:05.102373+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:06.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:06 smithi078 bash[23704]: cluster 2023-07-29T17:03:05.102373+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:06.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:06 smithi078 bash[28259]: cluster 2023-07-29T17:03:05.102373+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:07.221 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:03:08.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:08 smithi129 bash[23526]: cluster 2023-07-29T17:03:07.103030+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:08.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:08 smithi078 bash[23704]: cluster 2023-07-29T17:03:07.103030+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:08.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:08 smithi078 bash[28259]: cluster 2023-07-29T17:03:07.103030+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:10.164 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:03:10.164 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"fsid":"eb1107b2-2e30-11ee-9b36-001a4aab830c","created":"2023-07-29T16:57:39.388333+0000","modified":"2023-07-29T17:03:01.211389+0000","last_up_change":"2023-07-29T17:02:59.206615+0000","last_in_change":"2023-07-29T17:02:40.079717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-29T17:01:11.665187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e7a0e08d-36b3-407d-9d06-6a916d064a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6803","nonce":3465957458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6805","nonce":3465957458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6809","nonce":3465957458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6807","nonce":3465957458}]},"public_addr":"172.21.15.78:6803/3465957458","cluster_addr":"172.21.15.78:6805/3465957458","heartbeat_back_addr":"172.21.15.78:6809/3465957458","heartbeat_front_addr":"172.21.15.78:6807/3465957458","state":["exists","up"]},{"osd":1,"uuid":"567bfd6c-b9b4-4b44-994a-fa00e3cd0703","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6811","nonce":3824985047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6813","nonce":3824985047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6817","nonce":3824985047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6815","nonce":3824985047}]},"public_addr":"172.21.15.78:6811/3824985047","cluster_addr":"172.21.15.78:6813/3824985047","heartbeat_back_addr":"172.21.15.78:6817/3824985047","heartbeat_front_addr":"172.21.15.78:6815/3824985047","state":["exists","up"]},{"osd":2,"uuid":"a0714ad7-47d8-4b54-bcb4-80ebe0a96310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6819","nonce":3414832819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6821","nonce":3414832819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6825","nonce":3414832819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6823","nonce":3414832819}]},"public_addr":"172.21.15.78:6819/3414832819","cluster_addr":"172.21.15.78:6821/3414832819","heartbeat_back_addr":"172.21.15.78:6825/3414832819","heartbeat_front_addr":"172.21.15.78:6823/3414832819","state":["exists","up"]},{"osd":3,"uuid":"eba2c490-d3d9-4b10-9d86-5cccb0bdad5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6827","nonce":4175854573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6829","nonce":4175854573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6833","nonce":4175854573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6831","nonce":4175854573}]},"public_addr":"172.21.15.78:6827/4175854573","cluster_addr":"172.21.15.78:6829/4175854573","heartbeat_back_addr":"172.21.15.78:6833/4175854573","heartbeat_front_addr":"172.21.15.78:6831/4175854573","state":["exists","up"]},{"osd":4,"uuid":"678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6801","nonce":1329658569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6803","nonce":1329658569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6807","nonce":1329658569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6805","nonce":1329658569}]},"public_addr":"172.21.15.129:6801/1329658569","cluster_addr":"172.21.15.129:6803/1329658569","heartbeat_back_addr":"172.21.15.129:6807/1329658569","heartbeat_front_addr":"172.21.15.129:6805/1329658569","state":["exists","up"]},{"osd":5,"uuid":"fd04f2c7-9386-4d16-9f41-bd785f3eab89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6809","nonce":2755329734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6811","nonce":2755329734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6815","nonce":2755329734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6813","nonce":2755329734}]},"public_addr":"172.21.15.129:6809/2755329734","cluster_addr":"172.21.15.129:6811/2755329734","heartbeat_back_addr":"172.21.15.129:6815/2755329734","heartbeat_front_addr":"172.21.15.129:6813/2755329734","state":["exists","up"]},{"osd":6,"uuid":"392bb467-d2ee-441f-86f0-79e664b204ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6817","nonce":3214745648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6819","nonce":3214745648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6823","nonce":3214745648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6821","nonce":3214745648}]},"public_addr":"172.21.15.129:6817/3214745648","cluster_addr":"172.21.15.129:6819/3214745648","heartbeat_back_addr":"172.21.15.129:6823/3214745648","heartbeat_front_addr":"172.21.15.129:6821/3214745648","state":["exists","up"]},{"osd":7,"uuid":"690a3c3e-18d4-493f-a3fa-176f897a16d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6825","nonce":815409209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6827","nonce":815409209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6831","nonce":815409209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6829","nonce":815409209}]},"public_addr":"172.21.15.129:6825/815409209","cluster_addr":"172.21.15.129:6827/815409209","heartbeat_back_addr":"172.21.15.129:6831/815409209","heartbeat_front_addr":"172.21.15.129:6829/815409209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:03.352303+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:28.798523+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:55.137209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:01:19.513969+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:01:43.252260+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:07.859256+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:35.096547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:57.986054+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2788350456":"2023-07-30T16:58:34.973336+0000","172.21.15.78:0/165502057":"2023-07-30T16:58:34.973336+0000","172.21.15.78:6800/1993945665":"2023-07-30T16:58:34.973336+0000","172.21.15.78:0/4261163805":"2023-07-30T16:58:06.837056+0000","172.21.15.78:6800/3474529770":"2023-07-30T16:58:06.837056+0000","172.21.15.78:6801/1993945665":"2023-07-30T16:58:34.973336+0000","172.21.15.78:6801/3474529770":"2023-07-30T16:58:06.837056+0000","172.21.15.78:0/2206623442":"2023-07-30T16:58:06.837056+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T17:03:10.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:10 smithi129 bash[23526]: cluster 2023-07-29T17:03:09.103674+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:10.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:10 smithi078 bash[28259]: cluster 2023-07-29T17:03:09.103674+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:10.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:10 smithi078 bash[23704]: cluster 2023-07-29T17:03:09.103674+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:10.851 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-29T17:01:11.665187+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2023-07-29T17:03:10.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2023-07-29T17:03:11.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:11 smithi129 bash[23526]: audit 2023-07-29T17:03:10.159045+0000 mon.a (mon.0) 471 : audit [DBG] from='client.? 172.21.15.78:0/211171777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:03:11.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:11 smithi078 bash[23704]: audit 2023-07-29T17:03:10.159045+0000 mon.a (mon.0) 471 : audit [DBG] from='client.? 172.21.15.78:0/211171777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:03:11.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:11 smithi078 bash[28259]: audit 2023-07-29T17:03:10.159045+0000 mon.a (mon.0) 471 : audit [DBG] from='client.? 172.21.15.78:0/211171777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:03:12.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:12 smithi129 bash[23526]: cluster 2023-07-29T17:03:11.104292+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:12.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:12 smithi078 bash[23704]: cluster 2023-07-29T17:03:11.104292+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:12.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:12 smithi078 bash[28259]: cluster 2023-07-29T17:03:11.104292+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:13.593 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:03:14.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:14 smithi129 bash[23526]: cluster 2023-07-29T17:03:13.104866+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:14.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:14 smithi078 bash[23704]: cluster 2023-07-29T17:03:13.104866+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:14.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:14 smithi078 bash[28259]: cluster 2023-07-29T17:03:13.104866+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:16.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:16 smithi078 bash[23704]: cluster 2023-07-29T17:03:15.105444+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:16.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:16 smithi078 bash[28259]: cluster 2023-07-29T17:03:15.105444+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:16.656 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-29T17:03:16.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:16 smithi129 bash[23526]: cluster 2023-07-29T17:03:15.105444+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:17.360 INFO:tasks.cephadm:Adding prometheus.a on smithi129 2023-07-29T17:03:17.361 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch apply prometheus '1;smithi129=a' 2023-07-29T17:03:17.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:17 smithi078 bash[28259]: audit 2023-07-29T17:03:16.652071+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.21.15.78:0/1087554654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-07-29T17:03:17.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:17 smithi078 bash[23704]: audit 2023-07-29T17:03:16.652071+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.21.15.78:0/1087554654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-07-29T17:03:17.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:17 smithi129 bash[23526]: audit 2023-07-29T17:03:16.652071+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.21.15.78:0/1087554654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-07-29T17:03:18.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:18 smithi078 bash[23704]: cluster 2023-07-29T17:03:17.106049+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:18.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:18 smithi078 bash[28259]: cluster 2023-07-29T17:03:17.106049+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:18.490 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:18 smithi129 bash[23526]: cluster 2023-07-29T17:03:17.106049+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:20.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:20 smithi129 bash[23526]: cluster 2023-07-29T17:03:19.106586+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:20.460 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled prometheus update... 2023-07-29T17:03:20.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:20 smithi078 bash[23704]: cluster 2023-07-29T17:03:19.106586+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:20.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:20 smithi078 bash[28259]: cluster 2023-07-29T17:03:19.106586+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:21.164 DEBUG:teuthology.orchestra.run.smithi129:prometheus.a> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service 2023-07-29T17:03:21.166 INFO:tasks.cephadm:Adding node-exporter.a on smithi078 2023-07-29T17:03:21.166 INFO:tasks.cephadm:Adding node-exporter.b on smithi129 2023-07-29T17:03:21.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch apply node-exporter '2;smithi078=a;smithi129=b' 2023-07-29T17:03:21.507 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:03:21.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: audit 2023-07-29T17:03:20.446502+0000 mgr.y (mgr.14144) 188 : audit [DBG] from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:21.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: cephadm 2023-07-29T17:03:20.448004+0000 mgr.y (mgr.14144) 189 : cephadm [INF] Saving service prometheus spec with placement smithi129=a;count:1 2023-07-29T17:03:21.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: audit 2023-07-29T17:03:20.455157+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:21.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: audit 2023-07-29T17:03:20.456808+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:03:21.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: audit 2023-07-29T17:03:20.469202+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:21.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: cephadm 2023-07-29T17:03:20.486766+0000 mgr.y (mgr.14144) 190 : cephadm [INF] Deploying daemon prometheus.a on smithi129 2023-07-29T17:03:21.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:21 smithi129 bash[23526]: cluster 2023-07-29T17:03:21.107206+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:21.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: audit 2023-07-29T17:03:20.446502+0000 mgr.y (mgr.14144) 188 : audit [DBG] from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:21.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: cephadm 2023-07-29T17:03:20.448004+0000 mgr.y (mgr.14144) 189 : cephadm [INF] Saving service prometheus spec with placement smithi129=a;count:1 2023-07-29T17:03:21.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: audit 2023-07-29T17:03:20.455157+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:21.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: audit 2023-07-29T17:03:20.456808+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:03:21.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: audit 2023-07-29T17:03:20.469202+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:21.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: cephadm 2023-07-29T17:03:20.486766+0000 mgr.y (mgr.14144) 190 : cephadm [INF] Deploying daemon prometheus.a on smithi129 2023-07-29T17:03:21.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[23704]: cluster 2023-07-29T17:03:21.107206+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:21.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: audit 2023-07-29T17:03:20.446502+0000 mgr.y (mgr.14144) 188 : audit [DBG] from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:21.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: cephadm 2023-07-29T17:03:20.448004+0000 mgr.y (mgr.14144) 189 : cephadm [INF] Saving service prometheus spec with placement smithi129=a;count:1 2023-07-29T17:03:21.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: audit 2023-07-29T17:03:20.455157+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:21.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: audit 2023-07-29T17:03:20.456808+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:03:21.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: audit 2023-07-29T17:03:20.469202+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:21.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: cephadm 2023-07-29T17:03:20.486766+0000 mgr.y (mgr.14144) 190 : cephadm [INF] Deploying daemon prometheus.a on smithi129 2023-07-29T17:03:21.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:21 smithi078 bash[28259]: cluster 2023-07-29T17:03:21.107206+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:24.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:24 smithi078 bash[23704]: cluster 2023-07-29T17:03:23.107886+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:24.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:24 smithi078 bash[28259]: cluster 2023-07-29T17:03:23.107886+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:24.524 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:24 smithi129 bash[23526]: cluster 2023-07-29T17:03:23.107886+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:26.555 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:26 smithi129 bash[23526]: cluster 2023-07-29T17:03:25.108529+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:26.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:26 smithi078 bash[23704]: cluster 2023-07-29T17:03:25.108529+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:26.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:26 smithi078 bash[28259]: cluster 2023-07-29T17:03:25.108529+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:26.974 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled node-exporter update... 2023-07-29T17:03:27.903 DEBUG:teuthology.orchestra.run.smithi078:node-exporter.a> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service 2023-07-29T17:03:27.906 DEBUG:teuthology.orchestra.run.smithi129:node-exporter.b> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service 2023-07-29T17:03:27.908 INFO:tasks.cephadm:Adding alertmanager.a on smithi078 2023-07-29T17:03:27.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch apply alertmanager '1;smithi078=a' 2023-07-29T17:03:28.015 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:03:28.159 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:27 smithi129 bash[23526]: audit 2023-07-29T17:03:26.960268+0000 mgr.y (mgr.14144) 194 : audit [DBG] from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi078=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:28.160 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:27 smithi129 bash[23526]: cephadm 2023-07-29T17:03:26.961843+0000 mgr.y (mgr.14144) 195 : cephadm [INF] Saving service node-exporter spec with placement smithi078=a;smithi129=b;count:2 2023-07-29T17:03:28.160 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:27 smithi129 bash[23526]: audit 2023-07-29T17:03:26.968828+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:28.160 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:27 smithi129 bash[23526]: cluster 2023-07-29T17:03:27.109151+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:28.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[23704]: audit 2023-07-29T17:03:26.960268+0000 mgr.y (mgr.14144) 194 : audit [DBG] from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi078=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:28.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[23704]: cephadm 2023-07-29T17:03:26.961843+0000 mgr.y (mgr.14144) 195 : cephadm [INF] Saving service node-exporter spec with placement smithi078=a;smithi129=b;count:2 2023-07-29T17:03:28.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[23704]: audit 2023-07-29T17:03:26.968828+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:28.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[23704]: cluster 2023-07-29T17:03:27.109151+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:28.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[28259]: audit 2023-07-29T17:03:26.960268+0000 mgr.y (mgr.14144) 194 : audit [DBG] from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi078=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:28.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[28259]: cephadm 2023-07-29T17:03:26.961843+0000 mgr.y (mgr.14144) 195 : cephadm [INF] Saving service node-exporter spec with placement smithi078=a;smithi129=b;count:2 2023-07-29T17:03:28.177 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[28259]: audit 2023-07-29T17:03:26.968828+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:28.177 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:27 smithi078 bash[28259]: cluster 2023-07-29T17:03:27.109151+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:28.230 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:03:30.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:30 smithi078 bash[23704]: cluster 2023-07-29T17:03:29.109723+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:30.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:30 smithi078 bash[28259]: cluster 2023-07-29T17:03:29.109723+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:30.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:30 smithi129 bash[23526]: cluster 2023-07-29T17:03:29.109723+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:32.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:32 smithi129 bash[23526]: cluster 2023-07-29T17:03:31.110313+0000 mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:32.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:32 smithi078 bash[23704]: cluster 2023-07-29T17:03:31.110313+0000 mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:32.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:32 smithi078 bash[28259]: cluster 2023-07-29T17:03:31.110313+0000 mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:33.665 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled alertmanager update... 2023-07-29T17:03:34.606 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:34 smithi129 bash[23526]: cluster 2023-07-29T17:03:33.110885+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:34.607 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:34 smithi129 bash[23526]: audit 2023-07-29T17:03:33.659740+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:34.608 DEBUG:teuthology.orchestra.run.smithi078:alertmanager.a> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@alertmanager.a.service 2023-07-29T17:03:34.611 INFO:tasks.cephadm:Adding grafana.a on smithi129 2023-07-29T17:03:34.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph orch apply grafana '1;smithi129=a' 2023-07-29T17:03:34.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:34 smithi078 bash[23704]: cluster 2023-07-29T17:03:33.110885+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:34.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:34 smithi078 bash[23704]: audit 2023-07-29T17:03:33.659740+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:34.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:34 smithi078 bash[28259]: cluster 2023-07-29T17:03:33.110885+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:34.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:34 smithi078 bash[28259]: audit 2023-07-29T17:03:33.659740+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:34.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:03:35.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:35 smithi129 bash[23526]: audit 2023-07-29T17:03:33.649892+0000 mgr.y (mgr.14144) 200 : audit [DBG] from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi078=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:35.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:35 smithi129 bash[23526]: cephadm 2023-07-29T17:03:33.651433+0000 mgr.y (mgr.14144) 201 : cephadm [INF] Saving service alertmanager spec with placement smithi078=a;count:1 2023-07-29T17:03:35.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:35 smithi078 bash[23704]: audit 2023-07-29T17:03:33.649892+0000 mgr.y (mgr.14144) 200 : audit [DBG] from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi078=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:35.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:35 smithi078 bash[23704]: cephadm 2023-07-29T17:03:33.651433+0000 mgr.y (mgr.14144) 201 : cephadm [INF] Saving service alertmanager spec with placement smithi078=a;count:1 2023-07-29T17:03:35.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:35 smithi078 bash[28259]: audit 2023-07-29T17:03:33.649892+0000 mgr.y (mgr.14144) 200 : audit [DBG] from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi078=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:35.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:35 smithi078 bash[28259]: cephadm 2023-07-29T17:03:33.651433+0000 mgr.y (mgr.14144) 201 : cephadm [INF] Saving service alertmanager spec with placement smithi078=a;count:1 2023-07-29T17:03:36.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:36 smithi129 bash[23526]: cluster 2023-07-29T17:03:35.111456+0000 mgr.y (mgr.14144) 202 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:36.246 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:36 smithi129 bash[23526]: audit 2023-07-29T17:03:35.248367+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:03:36.246 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:36 smithi129 bash[23526]: audit 2023-07-29T17:03:35.251275+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:03:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:36 smithi078 bash[23704]: cluster 2023-07-29T17:03:35.111456+0000 mgr.y (mgr.14144) 202 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:36 smithi078 bash[23704]: audit 2023-07-29T17:03:35.248367+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:03:36.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:36 smithi078 bash[23704]: audit 2023-07-29T17:03:35.251275+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:03:36.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:36 smithi078 bash[28259]: cluster 2023-07-29T17:03:35.111456+0000 mgr.y (mgr.14144) 202 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:36.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:36 smithi078 bash[28259]: audit 2023-07-29T17:03:35.248367+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:03:36.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:36 smithi078 bash[28259]: audit 2023-07-29T17:03:35.251275+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:03:37.909 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:37 smithi129 bash[23526]: audit 2023-07-29T17:03:36.430525+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:37.910 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:37 smithi129 bash[23526]: audit 2023-07-29T17:03:36.432258+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:03:37.910 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:37 smithi129 bash[23526]: cluster 2023-07-29T17:03:37.112116+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:37.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:37 smithi078 bash[23704]: audit 2023-07-29T17:03:36.430525+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:37.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:37 smithi078 bash[23704]: audit 2023-07-29T17:03:36.432258+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:03:37.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:37 smithi078 bash[23704]: cluster 2023-07-29T17:03:37.112116+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:37.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:37 smithi078 bash[28259]: audit 2023-07-29T17:03:36.430525+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:37.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:37 smithi078 bash[28259]: audit 2023-07-29T17:03:36.432258+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:03:37.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:37 smithi078 bash[28259]: cluster 2023-07-29T17:03:37.112116+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:38.856 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:03:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:03:38.625Z caller=main.go:646 msg="Server is ready to receive web requests." 2023-07-29T17:03:39.275 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled grafana update... 2023-07-29T17:03:40.097 DEBUG:teuthology.orchestra.run.smithi129:grafana.a> sudo journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@grafana.a.service 2023-07-29T17:03:40.099 INFO:tasks.cephadm:Setting up client nodes... 2023-07-29T17:03:40.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-29T17:03:40.176 INFO:journalctl@ceph.grafana.a.smithi129.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-29T17:03:40.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:40 smithi129 bash[23526]: cluster 2023-07-29T17:03:39.112701+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:40.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:40 smithi129 bash[23526]: audit 2023-07-29T17:03:39.269890+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:40.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:40 smithi129 bash[23526]: audit 2023-07-29T17:03:39.965794+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:40.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:40 smithi078 bash[23704]: cluster 2023-07-29T17:03:39.112701+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:40.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:40 smithi078 bash[23704]: audit 2023-07-29T17:03:39.269890+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:40.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:40 smithi078 bash[23704]: audit 2023-07-29T17:03:39.965794+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:40.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:40 smithi078 bash[28259]: cluster 2023-07-29T17:03:39.112701+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:40.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:40 smithi078 bash[28259]: audit 2023-07-29T17:03:39.269890+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:40.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:40 smithi078 bash[28259]: audit 2023-07-29T17:03:39.965794+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:40 smithi129 systemd[1]: Started Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:03:41.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:40 smithi129 bash[37376]: Unable to find image 'prom/node-exporter:v0.18.1' locally 2023-07-29T17:03:41.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[23704]: audit 2023-07-29T17:03:39.261220+0000 mgr.y (mgr.14144) 205 : audit [DBG] from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:41.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[23704]: cephadm 2023-07-29T17:03:39.262669+0000 mgr.y (mgr.14144) 206 : cephadm [INF] Saving service grafana spec with placement smithi129=a;count:1 2023-07-29T17:03:41.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[23704]: audit 2023-07-29T17:03:40.215255+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[23704]: audit 2023-07-29T17:03:40.917908+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[28259]: audit 2023-07-29T17:03:39.261220+0000 mgr.y (mgr.14144) 205 : audit [DBG] from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:41.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[28259]: cephadm 2023-07-29T17:03:39.262669+0000 mgr.y (mgr.14144) 206 : cephadm [INF] Saving service grafana spec with placement smithi129=a;count:1 2023-07-29T17:03:41.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[28259]: audit 2023-07-29T17:03:40.215255+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[28259]: audit 2023-07-29T17:03:40.917908+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[23526]: audit 2023-07-29T17:03:39.261220+0000 mgr.y (mgr.14144) 205 : audit [DBG] from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:03:41.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[23526]: cephadm 2023-07-29T17:03:39.262669+0000 mgr.y (mgr.14144) 206 : cephadm [INF] Saving service grafana spec with placement smithi129=a;count:1 2023-07-29T17:03:41.634 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[23526]: audit 2023-07-29T17:03:40.215255+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.634 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[23526]: audit 2023-07-29T17:03:40.917908+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:41.675 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:41 smithi078 systemd[1]: Started Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:03:41.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: v0.18.1: Pulling from prom/node-exporter 2023-07-29T17:03:41.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 49a2d53aa1af: Pulling fs layer 2023-07-29T17:03:41.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 3589a6efd9ce: Pulling fs layer 2023-07-29T17:03:41.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 190160031744: Pulling fs layer 2023-07-29T17:03:41.925 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:41 smithi078 bash[41279]: Unable to find image 'prom/node-exporter:v0.18.1' locally 2023-07-29T17:03:42.159 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 3589a6efd9ce: Download complete 2023-07-29T17:03:42.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 49a2d53aa1af: Verifying Checksum 2023-07-29T17:03:42.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 49a2d53aa1af: Download complete 2023-07-29T17:03:42.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 190160031744: Verifying Checksum 2023-07-29T17:03:42.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:41 smithi129 bash[37376]: 190160031744: Download complete 2023-07-29T17:03:42.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[37376]: 49a2d53aa1af: Pull complete 2023-07-29T17:03:42.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[28259]: cephadm 2023-07-29T17:03:40.218184+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Deploying daemon node-exporter.b on smithi129 2023-07-29T17:03:42.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[28259]: cephadm 2023-07-29T17:03:40.919616+0000 mgr.y (mgr.14144) 208 : cephadm [INF] Deploying daemon node-exporter.a on smithi078 2023-07-29T17:03:42.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[28259]: cluster 2023-07-29T17:03:41.113425+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:42.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[28259]: audit 2023-07-29T17:03:41.657930+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:42.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[23704]: cephadm 2023-07-29T17:03:40.218184+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Deploying daemon node-exporter.b on smithi129 2023-07-29T17:03:42.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[23704]: cephadm 2023-07-29T17:03:40.919616+0000 mgr.y (mgr.14144) 208 : cephadm [INF] Deploying daemon node-exporter.a on smithi078 2023-07-29T17:03:42.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[23704]: cluster 2023-07-29T17:03:41.113425+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:42.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[23704]: audit 2023-07-29T17:03:41.657930+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:42.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[23526]: cephadm 2023-07-29T17:03:40.218184+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Deploying daemon node-exporter.b on smithi129 2023-07-29T17:03:42.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[23526]: cephadm 2023-07-29T17:03:40.919616+0000 mgr.y (mgr.14144) 208 : cephadm [INF] Deploying daemon node-exporter.a on smithi078 2023-07-29T17:03:42.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[23526]: cluster 2023-07-29T17:03:41.113425+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:42.502 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[23526]: audit 2023-07-29T17:03:41.657930+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:42.861 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[37376]: 3589a6efd9ce: Pull complete 2023-07-29T17:03:42.925 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:42 smithi078 bash[41279]: v0.18.1: Pulling from prom/node-exporter 2023-07-29T17:03:43.159 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:42 smithi129 bash[37376]: 190160031744: Pull complete 2023-07-29T17:03:43.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:43 smithi129 bash[37376]: Digest: sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee 2023-07-29T17:03:43.160 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:43 smithi129 bash[37376]: Status: Downloaded newer image for prom/node-exporter:v0.18.1 2023-07-29T17:03:43.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:43 smithi129 bash[23526]: cephadm 2023-07-29T17:03:41.664351+0000 mgr.y (mgr.14144) 210 : cephadm [INF] Deploying daemon alertmanager.a on smithi078 2023-07-29T17:03:43.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[23704]: cephadm 2023-07-29T17:03:41.664351+0000 mgr.y (mgr.14144) 210 : cephadm [INF] Deploying daemon alertmanager.a on smithi078 2023-07-29T17:03:43.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[28259]: cephadm 2023-07-29T17:03:41.664351+0000 mgr.y (mgr.14144) 210 : cephadm [INF] Deploying daemon alertmanager.a on smithi078 2023-07-29T17:03:43.480 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 49a2d53aa1af: Pulling fs layer 2023-07-29T17:03:43.480 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 3589a6efd9ce: Pulling fs layer 2023-07-29T17:03:43.480 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 190160031744: Pulling fs layer 2023-07-29T17:03:43.480 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 3589a6efd9ce: Verifying Checksum 2023-07-29T17:03:43.481 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 3589a6efd9ce: Download complete 2023-07-29T17:03:43.481 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 49a2d53aa1af: Verifying Checksum 2023-07-29T17:03:43.481 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 49a2d53aa1af: Download complete 2023-07-29T17:03:43.792 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 190160031744: Verifying Checksum 2023-07-29T17:03:43.793 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 190160031744: Download complete 2023-07-29T17:03:43.793 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:43 smithi078 bash[41279]: 49a2d53aa1af: Pull complete 2023-07-29T17:03:44.277 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2023-07-29T17:03:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBwRsVkjtDzDxAAcN6fKhc2nSym6N2DlAS/1g== 2023-07-29T17:03:44.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:44 smithi078 bash[23704]: cluster 2023-07-29T17:03:43.114066+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:44.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:44 smithi078 bash[28259]: cluster 2023-07-29T17:03:43.114066+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:44.426 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:44 smithi078 bash[41279]: 3589a6efd9ce: Pull complete 2023-07-29T17:03:44.494 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[23526]: cluster 2023-07-29T17:03:43.114066+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:44.910 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-29T17:03:44.911 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-29T17:03:44.911 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-29T17:03:44.911 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-29T17:03:44.911 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-29T17:03:44.911 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-29T17:03:44.911 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-29T17:03:44.912 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-29T17:03:44.912 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-29T17:03:44.912 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-29T17:03:44.912 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-29T17:03:44.912 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-29T17:03:44.913 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-29T17:03:44.914 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-29T17:03:44.915 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:03:44 smithi129 bash[37376]: time="2023-07-29T17:03:44Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-29T17:03:44.925 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:44 smithi078 bash[41279]: 190160031744: Pull complete 2023-07-29T17:03:45.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:45 smithi129 bash[23526]: audit 2023-07-29T17:03:44.266984+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.78:0/1328431086' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:45.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:45 smithi129 bash[23526]: audit 2023-07-29T17:03:44.267429+0000 mon.a (mon.0) 487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:45.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:45 smithi129 bash[23526]: audit 2023-07-29T17:03:44.272878+0000 mon.a (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T17:03:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[23704]: audit 2023-07-29T17:03:44.266984+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.78:0/1328431086' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[23704]: audit 2023-07-29T17:03:44.267429+0000 mon.a (mon.0) 487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[23704]: audit 2023-07-29T17:03:44.272878+0000 mon.a (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T17:03:45.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[28259]: audit 2023-07-29T17:03:44.266984+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.78:0/1328431086' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:45.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[28259]: audit 2023-07-29T17:03:44.267429+0000 mon.a (mon.0) 487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:45.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[28259]: audit 2023-07-29T17:03:44.272878+0000 mon.a (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T17:03:46.150 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-29T17:03:46.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-29T17:03:46.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-29T17:03:46.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-29T17:03:46.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:46 smithi078 bash[23704]: cluster 2023-07-29T17:03:45.114724+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:46.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:46 smithi078 bash[28259]: cluster 2023-07-29T17:03:45.114724+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:46.426 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:45 smithi078 bash[41279]: Digest: sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee 2023-07-29T17:03:46.427 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:46 smithi078 bash[41279]: Status: Downloaded newer image for prom/node-exporter:v0.18.1 2023-07-29T17:03:46.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:46 smithi129 bash[23526]: cluster 2023-07-29T17:03:45.114724+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:48.115 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-29T17:03:48.116 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-29T17:03:48.116 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-29T17:03:48.116 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-29T17:03:48.117 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-29T17:03:48.117 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-29T17:03:48.117 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-29T17:03:48.117 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-29T17:03:48.117 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-29T17:03:48.118 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-29T17:03:48.118 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-29T17:03:48.118 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-29T17:03:48.118 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-29T17:03:48.118 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-29T17:03:48.119 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-29T17:03:48.120 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-29T17:03:48.120 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-29T17:03:48.120 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-29T17:03:48.120 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-29T17:03:48.120 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-29T17:03:48.121 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-29T17:03:48.121 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-29T17:03:48.121 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-29T17:03:48.121 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-29T17:03:48.121 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-29T17:03:48.122 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-29T17:03:48.122 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-29T17:03:48.122 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-29T17:03:48.122 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-29T17:03:48.123 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-29T17:03:48.123 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-29T17:03:48.123 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-29T17:03:48.123 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-29T17:03:48.123 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-29T17:03:48.123 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:03:47 smithi078 bash[41279]: time="2023-07-29T17:03:47Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-29T17:03:48.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:48 smithi078 bash[23704]: cluster 2023-07-29T17:03:47.115422+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:48.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:48 smithi078 bash[28259]: cluster 2023-07-29T17:03:47.115422+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:48.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:48 smithi129 bash[23526]: cluster 2023-07-29T17:03:47.115422+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:49.610 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2023-07-29T17:03:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB1RsVkpWSnIxAALudfjQvGVWYwxCWGzC/8VQ== 2023-07-29T17:03:50.274 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-07-29T17:03:50.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-07-29T17:03:50.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-07-29T17:03:50.299 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-29T17:03:50.300 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-29T17:03:50.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph mgr dump --format=json 2023-07-29T17:03:50.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[28259]: cluster 2023-07-29T17:03:49.116067+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:50.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[28259]: audit 2023-07-29T17:03:49.596599+0000 mon.b (mon.1) 24 : audit [INF] from='client.? 172.21.15.129:0/4144389455' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:50.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[28259]: audit 2023-07-29T17:03:49.597772+0000 mon.a (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:50.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[28259]: audit 2023-07-29T17:03:49.605203+0000 mon.a (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T17:03:50.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[23704]: cluster 2023-07-29T17:03:49.116067+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:50.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[23704]: audit 2023-07-29T17:03:49.596599+0000 mon.b (mon.1) 24 : audit [INF] from='client.? 172.21.15.129:0/4144389455' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:50.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[23704]: audit 2023-07-29T17:03:49.597772+0000 mon.a (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:50.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:50 smithi078 bash[23704]: audit 2023-07-29T17:03:49.605203+0000 mon.a (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T17:03:50.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:50 smithi129 bash[23526]: cluster 2023-07-29T17:03:49.116067+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:50.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:50 smithi129 bash[23526]: audit 2023-07-29T17:03:49.596599+0000 mon.b (mon.1) 24 : audit [INF] from='client.? 172.21.15.129:0/4144389455' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:50.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:50 smithi129 bash[23526]: audit 2023-07-29T17:03:49.597772+0000 mon.a (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T17:03:50.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:50 smithi129 bash[23526]: audit 2023-07-29T17:03:49.605203+0000 mon.a (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T17:03:52.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:52 smithi078 bash[23704]: cluster 2023-07-29T17:03:51.116713+0000 mgr.y (mgr.14144) 215 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:52.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:52 smithi078 bash[28259]: cluster 2023-07-29T17:03:51.116713+0000 mgr.y (mgr.14144) 215 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:52.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:52 smithi129 bash[23526]: cluster 2023-07-29T17:03:51.116713+0000 mgr.y (mgr.14144) 215 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:52.676 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:52 smithi078 systemd[1]: Started Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:03:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:03:53.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: audit 2023-07-29T17:03:52.376245+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: audit 2023-07-29T17:03:52.520047+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: audit 2023-07-29T17:03:52.527547+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: audit 2023-07-29T17:03:52.528520+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:03:53.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: audit 2023-07-29T17:03:52.529229+0000 mgr.y (mgr.14144) 216 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:03:53.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: audit 2023-07-29T17:03:52.535366+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: cephadm 2023-07-29T17:03:52.540151+0000 mgr.y (mgr.14144) 217 : cephadm [INF] Deploying daemon grafana.a on smithi129 2023-07-29T17:03:53.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:53 smithi129 bash[23526]: cluster 2023-07-29T17:03:53.117404+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:53.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: audit 2023-07-29T17:03:52.376245+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: audit 2023-07-29T17:03:52.520047+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: audit 2023-07-29T17:03:52.527547+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: audit 2023-07-29T17:03:52.528520+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:03:53.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: audit 2023-07-29T17:03:52.529229+0000 mgr.y (mgr.14144) 216 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:03:53.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: audit 2023-07-29T17:03:52.535366+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: cephadm 2023-07-29T17:03:52.540151+0000 mgr.y (mgr.14144) 217 : cephadm [INF] Deploying daemon grafana.a on smithi129 2023-07-29T17:03:53.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[23704]: cluster 2023-07-29T17:03:53.117404+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:53.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: audit 2023-07-29T17:03:52.376245+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: audit 2023-07-29T17:03:52.520047+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: audit 2023-07-29T17:03:52.527547+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: audit 2023-07-29T17:03:52.528520+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:03:53.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: audit 2023-07-29T17:03:52.529229+0000 mgr.y (mgr.14144) 216 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:03:53.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: audit 2023-07-29T17:03:52.535366+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:03:53.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: cephadm 2023-07-29T17:03:52.540151+0000 mgr.y (mgr.14144) 217 : cephadm [INF] Deploying daemon grafana.a on smithi129 2023-07-29T17:03:53.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:53 smithi078 bash[28259]: cluster 2023-07-29T17:03:53.117404+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:54.606 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.453Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2023-07-29T17:03:54.606 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.454Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2023-07-29T17:03:54.606 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.456Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.78 port=9094 2023-07-29T17:03:54.606 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.458Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-29T17:03:54.607 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.507Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:03:54.607 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.507Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:03:54.607 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:54 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:54.512Z caller=main.go:497 msg=Listening address=:9093 2023-07-29T17:03:56.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:56 smithi078 bash[28259]: cluster 2023-07-29T17:03:55.118056+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:56.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:56 smithi078 bash[23704]: cluster 2023-07-29T17:03:55.118056+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:56.578 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:03:56.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:56 smithi129 bash[23526]: cluster 2023-07-29T17:03:55.118056+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:56.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:03:56 smithi078 bash[41885]: level=info ts=2023-07-29T17:03:56.458Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000161128s 2023-07-29T17:03:57.249 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":14,"active_gid":14144,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":2098220507},{"type":"v1","addr":"172.21.15.78:6801","nonce":2098220507}]},"active_addr":"172.21.15.78:6801/2098220507","active_change":"2023-07-29T16:58:34.973912+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi078.front.sepia.ceph.com:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":4132963352}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":144224075}]}]}} 2023-07-29T17:03:57.255 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-29T17:03:57.255 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-29T17:03:57.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd dump --format=json 2023-07-29T17:03:57.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:57 smithi129 bash[23526]: audit 2023-07-29T17:03:56.563925+0000 mon.a (mon.0) 496 : audit [DBG] from='client.? 172.21.15.78:0/98558629' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-29T17:03:57.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:57 smithi078 bash[23704]: audit 2023-07-29T17:03:56.563925+0000 mon.a (mon.0) 496 : audit [DBG] from='client.? 172.21.15.78:0/98558629' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-29T17:03:57.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:57 smithi078 bash[28259]: audit 2023-07-29T17:03:56.563925+0000 mon.a (mon.0) 496 : audit [DBG] from='client.? 172.21.15.78:0/98558629' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-29T17:03:58.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:03:58 smithi129 bash[23526]: cluster 2023-07-29T17:03:57.118661+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:58.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:03:58 smithi078 bash[28259]: cluster 2023-07-29T17:03:57.118661+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:03:58.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:03:58 smithi078 bash[23704]: cluster 2023-07-29T17:03:57.118661+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:00.000 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:00.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:00 smithi078 bash[23704]: cluster 2023-07-29T17:03:59.119245+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:00.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:00 smithi078 bash[28259]: cluster 2023-07-29T17:03:59.119245+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:00.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:00 smithi129 bash[23526]: cluster 2023-07-29T17:03:59.119245+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:02 smithi078 bash[23704]: cluster 2023-07-29T17:04:01.119866+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:02 smithi078 bash[28259]: cluster 2023-07-29T17:04:01.119866+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:02.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:02 smithi129 bash[23526]: cluster 2023-07-29T17:04:01.119866+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:02.906 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:04:02.906 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"fsid":"eb1107b2-2e30-11ee-9b36-001a4aab830c","created":"2023-07-29T16:57:39.388333+0000","modified":"2023-07-29T17:03:01.211389+0000","last_up_change":"2023-07-29T17:02:59.206615+0000","last_in_change":"2023-07-29T17:02:40.079717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-29T17:01:11.665187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e7a0e08d-36b3-407d-9d06-6a916d064a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6803","nonce":3465957458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6805","nonce":3465957458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6809","nonce":3465957458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6807","nonce":3465957458}]},"public_addr":"172.21.15.78:6803/3465957458","cluster_addr":"172.21.15.78:6805/3465957458","heartbeat_back_addr":"172.21.15.78:6809/3465957458","heartbeat_front_addr":"172.21.15.78:6807/3465957458","state":["exists","up"]},{"osd":1,"uuid":"567bfd6c-b9b4-4b44-994a-fa00e3cd0703","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6811","nonce":3824985047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6813","nonce":3824985047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6817","nonce":3824985047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6815","nonce":3824985047}]},"public_addr":"172.21.15.78:6811/3824985047","cluster_addr":"172.21.15.78:6813/3824985047","heartbeat_back_addr":"172.21.15.78:6817/3824985047","heartbeat_front_addr":"172.21.15.78:6815/3824985047","state":["exists","up"]},{"osd":2,"uuid":"a0714ad7-47d8-4b54-bcb4-80ebe0a96310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6819","nonce":3414832819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6821","nonce":3414832819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6825","nonce":3414832819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6823","nonce":3414832819}]},"public_addr":"172.21.15.78:6819/3414832819","cluster_addr":"172.21.15.78:6821/3414832819","heartbeat_back_addr":"172.21.15.78:6825/3414832819","heartbeat_front_addr":"172.21.15.78:6823/3414832819","state":["exists","up"]},{"osd":3,"uuid":"eba2c490-d3d9-4b10-9d86-5cccb0bdad5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6827","nonce":4175854573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6829","nonce":4175854573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6833","nonce":4175854573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6831","nonce":4175854573}]},"public_addr":"172.21.15.78:6827/4175854573","cluster_addr":"172.21.15.78:6829/4175854573","heartbeat_back_addr":"172.21.15.78:6833/4175854573","heartbeat_front_addr":"172.21.15.78:6831/4175854573","state":["exists","up"]},{"osd":4,"uuid":"678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6801","nonce":1329658569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6803","nonce":1329658569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6807","nonce":1329658569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6805","nonce":1329658569}]},"public_addr":"172.21.15.129:6801/1329658569","cluster_addr":"172.21.15.129:6803/1329658569","heartbeat_back_addr":"172.21.15.129:6807/1329658569","heartbeat_front_addr":"172.21.15.129:6805/1329658569","state":["exists","up"]},{"osd":5,"uuid":"fd04f2c7-9386-4d16-9f41-bd785f3eab89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6809","nonce":2755329734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6811","nonce":2755329734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6815","nonce":2755329734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6813","nonce":2755329734}]},"public_addr":"172.21.15.129:6809/2755329734","cluster_addr":"172.21.15.129:6811/2755329734","heartbeat_back_addr":"172.21.15.129:6815/2755329734","heartbeat_front_addr":"172.21.15.129:6813/2755329734","state":["exists","up"]},{"osd":6,"uuid":"392bb467-d2ee-441f-86f0-79e664b204ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6817","nonce":3214745648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6819","nonce":3214745648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6823","nonce":3214745648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6821","nonce":3214745648}]},"public_addr":"172.21.15.129:6817/3214745648","cluster_addr":"172.21.15.129:6819/3214745648","heartbeat_back_addr":"172.21.15.129:6823/3214745648","heartbeat_front_addr":"172.21.15.129:6821/3214745648","state":["exists","up"]},{"osd":7,"uuid":"690a3c3e-18d4-493f-a3fa-176f897a16d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6825","nonce":815409209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6827","nonce":815409209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6831","nonce":815409209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6829","nonce":815409209}]},"public_addr":"172.21.15.129:6825/815409209","cluster_addr":"172.21.15.129:6827/815409209","heartbeat_back_addr":"172.21.15.129:6831/815409209","heartbeat_front_addr":"172.21.15.129:6829/815409209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:03.352303+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:28.798523+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:55.137209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:01:19.513969+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:01:43.252260+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:07.859256+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:35.096547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:57.986054+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2788350456":"2023-07-30T16:58:34.973336+0000","172.21.15.78:0/165502057":"2023-07-30T16:58:34.973336+0000","172.21.15.78:6800/1993945665":"2023-07-30T16:58:34.973336+0000","172.21.15.78:0/4261163805":"2023-07-30T16:58:06.837056+0000","172.21.15.78:6800/3474529770":"2023-07-30T16:58:06.837056+0000","172.21.15.78:6801/1993945665":"2023-07-30T16:58:34.973336+0000","172.21.15.78:6801/3474529770":"2023-07-30T16:58:06.837056+0000","172.21.15.78:0/2206623442":"2023-07-30T16:58:06.837056+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T17:04:03.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:03 smithi078 bash[23704]: audit 2023-07-29T17:04:02.900065+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? 172.21.15.78:0/3459870954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:04:03.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:03 smithi078 bash[28259]: audit 2023-07-29T17:04:02.900065+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? 172.21.15.78:0/3459870954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:04:03.596 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-29T17:04:03.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd dump --format=json 2023-07-29T17:04:03.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:03 smithi129 bash[23526]: audit 2023-07-29T17:04:02.900065+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? 172.21.15.78:0/3459870954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:04:04.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:04 smithi078 bash[23704]: cluster 2023-07-29T17:04:03.120433+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:04.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:04 smithi078 bash[28259]: cluster 2023-07-29T17:04:03.120433+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:04.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:04 smithi129 bash[23526]: cluster 2023-07-29T17:04:03.120433+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:04.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:04 smithi078 bash[41885]: level=info ts=2023-07-29T17:04:04.459Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000941301s 2023-07-29T17:04:05.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:05 smithi129 bash[23526]: cluster 2023-07-29T17:04:05.121109+0000 mgr.y (mgr.14144) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:05.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:05 smithi078 bash[23704]: cluster 2023-07-29T17:04:05.121109+0000 mgr.y (mgr.14144) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:05.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:05 smithi078 bash[28259]: cluster 2023-07-29T17:04:05.121109+0000 mgr.y (mgr.14144) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:08.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:08 smithi129 bash[23526]: cluster 2023-07-29T17:04:07.121710+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:08.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:08 smithi078 bash[23704]: cluster 2023-07-29T17:04:07.121710+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:08.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:08 smithi078 bash[28259]: cluster 2023-07-29T17:04:07.121710+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:09.323 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:04:09.323 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"fsid":"eb1107b2-2e30-11ee-9b36-001a4aab830c","created":"2023-07-29T16:57:39.388333+0000","modified":"2023-07-29T17:03:01.211389+0000","last_up_change":"2023-07-29T17:02:59.206615+0000","last_in_change":"2023-07-29T17:02:40.079717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-29T17:01:11.665187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e7a0e08d-36b3-407d-9d06-6a916d064a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6803","nonce":3465957458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6805","nonce":3465957458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6809","nonce":3465957458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3465957458},{"type":"v1","addr":"172.21.15.78:6807","nonce":3465957458}]},"public_addr":"172.21.15.78:6803/3465957458","cluster_addr":"172.21.15.78:6805/3465957458","heartbeat_back_addr":"172.21.15.78:6809/3465957458","heartbeat_front_addr":"172.21.15.78:6807/3465957458","state":["exists","up"]},{"osd":1,"uuid":"567bfd6c-b9b4-4b44-994a-fa00e3cd0703","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6811","nonce":3824985047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6813","nonce":3824985047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6817","nonce":3824985047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3824985047},{"type":"v1","addr":"172.21.15.78:6815","nonce":3824985047}]},"public_addr":"172.21.15.78:6811/3824985047","cluster_addr":"172.21.15.78:6813/3824985047","heartbeat_back_addr":"172.21.15.78:6817/3824985047","heartbeat_front_addr":"172.21.15.78:6815/3824985047","state":["exists","up"]},{"osd":2,"uuid":"a0714ad7-47d8-4b54-bcb4-80ebe0a96310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6819","nonce":3414832819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6821","nonce":3414832819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6825","nonce":3414832819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3414832819},{"type":"v1","addr":"172.21.15.78:6823","nonce":3414832819}]},"public_addr":"172.21.15.78:6819/3414832819","cluster_addr":"172.21.15.78:6821/3414832819","heartbeat_back_addr":"172.21.15.78:6825/3414832819","heartbeat_front_addr":"172.21.15.78:6823/3414832819","state":["exists","up"]},{"osd":3,"uuid":"eba2c490-d3d9-4b10-9d86-5cccb0bdad5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6827","nonce":4175854573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6829","nonce":4175854573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6833","nonce":4175854573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":4175854573},{"type":"v1","addr":"172.21.15.78:6831","nonce":4175854573}]},"public_addr":"172.21.15.78:6827/4175854573","cluster_addr":"172.21.15.78:6829/4175854573","heartbeat_back_addr":"172.21.15.78:6833/4175854573","heartbeat_front_addr":"172.21.15.78:6831/4175854573","state":["exists","up"]},{"osd":4,"uuid":"678e0d9f-5d5d-4a5c-8c1b-3d3e942381d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6801","nonce":1329658569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6803","nonce":1329658569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6807","nonce":1329658569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1329658569},{"type":"v1","addr":"172.21.15.129:6805","nonce":1329658569}]},"public_addr":"172.21.15.129:6801/1329658569","cluster_addr":"172.21.15.129:6803/1329658569","heartbeat_back_addr":"172.21.15.129:6807/1329658569","heartbeat_front_addr":"172.21.15.129:6805/1329658569","state":["exists","up"]},{"osd":5,"uuid":"fd04f2c7-9386-4d16-9f41-bd785f3eab89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6809","nonce":2755329734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6811","nonce":2755329734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6815","nonce":2755329734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2755329734},{"type":"v1","addr":"172.21.15.129:6813","nonce":2755329734}]},"public_addr":"172.21.15.129:6809/2755329734","cluster_addr":"172.21.15.129:6811/2755329734","heartbeat_back_addr":"172.21.15.129:6815/2755329734","heartbeat_front_addr":"172.21.15.129:6813/2755329734","state":["exists","up"]},{"osd":6,"uuid":"392bb467-d2ee-441f-86f0-79e664b204ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6817","nonce":3214745648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6819","nonce":3214745648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6823","nonce":3214745648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3214745648},{"type":"v1","addr":"172.21.15.129:6821","nonce":3214745648}]},"public_addr":"172.21.15.129:6817/3214745648","cluster_addr":"172.21.15.129:6819/3214745648","heartbeat_back_addr":"172.21.15.129:6823/3214745648","heartbeat_front_addr":"172.21.15.129:6821/3214745648","state":["exists","up"]},{"osd":7,"uuid":"690a3c3e-18d4-493f-a3fa-176f897a16d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6825","nonce":815409209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6827","nonce":815409209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6831","nonce":815409209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":815409209},{"type":"v1","addr":"172.21.15.129:6829","nonce":815409209}]},"public_addr":"172.21.15.129:6825/815409209","cluster_addr":"172.21.15.129:6827/815409209","heartbeat_back_addr":"172.21.15.129:6831/815409209","heartbeat_front_addr":"172.21.15.129:6829/815409209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:03.352303+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:28.798523+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:00:55.137209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:01:19.513969+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:01:43.252260+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:07.859256+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:35.096547+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T17:02:57.986054+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2788350456":"2023-07-30T16:58:34.973336+0000","172.21.15.78:0/165502057":"2023-07-30T16:58:34.973336+0000","172.21.15.78:6800/1993945665":"2023-07-30T16:58:34.973336+0000","172.21.15.78:0/4261163805":"2023-07-30T16:58:06.837056+0000","172.21.15.78:6800/3474529770":"2023-07-30T16:58:06.837056+0000","172.21.15.78:6801/1993945665":"2023-07-30T16:58:34.973336+0000","172.21.15.78:6801/3474529770":"2023-07-30T16:58:06.837056+0000","172.21.15.78:0/2206623442":"2023-07-30T16:58:06.837056+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T17:04:10.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-29T17:04:10.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-29T17:04:10.029 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-29T17:04:10.029 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-29T17:04:10.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-29T17:04:10.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-29T17:04:10.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-29T17:04:10.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-29T17:04:10.211 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2023-07-29T17:04:10.211 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[23526]: audit 2023-07-29T17:04:08.993753+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:10.212 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[23526]: audit 2023-07-29T17:04:08.995023+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:10.212 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[23526]: cluster 2023-07-29T17:04:09.122173+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:10.212 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[23526]: audit 2023-07-29T17:04:09.316611+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? 172.21.15.78:0/3940286467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:04:10.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[28259]: audit 2023-07-29T17:04:08.993753+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:10.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[28259]: audit 2023-07-29T17:04:08.995023+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:10.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[28259]: cluster 2023-07-29T17:04:09.122173+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:10.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[28259]: audit 2023-07-29T17:04:09.316611+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? 172.21.15.78:0/3940286467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:04:10.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[23704]: audit 2023-07-29T17:04:08.993753+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:10.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[23704]: audit 2023-07-29T17:04:08.995023+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:10.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[23704]: cluster 2023-07-29T17:04:09.122173+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:10.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:10 smithi078 bash[23704]: audit 2023-07-29T17:04:09.316611+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? 172.21.15.78:0/3940286467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T17:04:10.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2023-07-29T17:04:10.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2023-07-29T17:04:10.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2023-07-29T17:04:10.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2023-07-29T17:04:10.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2023-07-29T17:04:10.857 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2023-07-29T17:04:10.857 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2023-07-29T17:04:10.857 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2023-07-29T17:04:10.858 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-29T17:04:10.858 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2023-07-29T17:04:11.150 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2023-07-29T17:04:11.151 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2023-07-29T17:04:11.151 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:10 smithi129 bash[37985]: t=2023-07-29T17:04:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2023-07-29T17:04:11.151 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2023-07-29T17:04:11.151 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2023-07-29T17:04:11.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2023-07-29T17:04:11.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2023-07-29T17:04:11.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2023-07-29T17:04:11.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2023-07-29T17:04:11.411 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2023-07-29T17:04:11.696 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2023-07-29T17:04:11.696 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2023-07-29T17:04:11.696 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2023-07-29T17:04:11.697 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2023-07-29T17:04:11.697 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2023-07-29T17:04:11.958 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2023-07-29T17:04:11.959 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2023-07-29T17:04:11.959 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2023-07-29T17:04:12.210 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[23526]: cluster 2023-07-29T17:04:11.122736+0000 mgr.y (mgr.14144) 227 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:12.210 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:11 smithi129 bash[37985]: t=2023-07-29T17:04:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2023-07-29T17:04:12.210 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2023-07-29T17:04:12.211 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2023-07-29T17:04:12.529 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2023-07-29T17:04:12.529 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2023-07-29T17:04:12.529 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2023-07-29T17:04:12.529 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2023-07-29T17:04:12.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:12 smithi078 bash[23704]: cluster 2023-07-29T17:04:11.122736+0000 mgr.y (mgr.14144) 227 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:12.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:12 smithi078 bash[28259]: cluster 2023-07-29T17:04:11.122736+0000 mgr.y (mgr.14144) 227 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:12.761 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.762 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.765 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.766 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.766 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.767 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.771 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:12.780 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2023-07-29T17:04:12.780 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-29T17:04:12.780 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2023-07-29T17:04:12.781 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2023-07-29T17:04:13.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2023-07-29T17:04:13.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2023-07-29T17:04:13.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:12 smithi129 bash[37985]: t=2023-07-29T17:04:12+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2023-07-29T17:04:13.409 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2023-07-29T17:04:13.409 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2023-07-29T17:04:13.409 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2023-07-29T17:04:13.409 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2023-07-29T17:04:13.659 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2023-07-29T17:04:13.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2023-07-29T17:04:13.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2023-07-29T17:04:13.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2023-07-29T17:04:13.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[23704]: audit 2023-07-29T17:04:12.663707+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[23704]: audit 2023-07-29T17:04:12.880285+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[23704]: audit 2023-07-29T17:04:12.887824+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[23704]: cephadm 2023-07-29T17:04:12.894251+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:04:13.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[23704]: cephadm 2023-07-29T17:04:12.900329+0000 mgr.y (mgr.14144) 229 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:04:13.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[23704]: cluster 2023-07-29T17:04:13.123345+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:13.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[28259]: audit 2023-07-29T17:04:12.663707+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[28259]: audit 2023-07-29T17:04:12.880285+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[28259]: audit 2023-07-29T17:04:12.887824+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[28259]: cephadm 2023-07-29T17:04:12.894251+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:04:13.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[28259]: cephadm 2023-07-29T17:04:12.900329+0000 mgr.y (mgr.14144) 229 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:04:13.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:13 smithi078 bash[28259]: cluster 2023-07-29T17:04:13.123345+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:13.987 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2023-07-29T17:04:13.988 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2023-07-29T17:04:13.988 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2023-07-29T17:04:13.988 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2023-07-29T17:04:13.988 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[23526]: audit 2023-07-29T17:04:12.663707+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.988 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[23526]: audit 2023-07-29T17:04:12.880285+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.989 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[23526]: audit 2023-07-29T17:04:12.887824+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:13.989 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[23526]: cephadm 2023-07-29T17:04:12.894251+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:04:13.989 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[23526]: cephadm 2023-07-29T17:04:12.900329+0000 mgr.y (mgr.14144) 229 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:04:13.989 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[23526]: cluster 2023-07-29T17:04:13.123345+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:14.247 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:13 smithi129 bash[37985]: t=2023-07-29T17:04:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2023-07-29T17:04:14.247 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2023-07-29T17:04:14.248 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2023-07-29T17:04:14.248 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2023-07-29T17:04:14.599 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2023-07-29T17:04:14.599 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2023-07-29T17:04:14.599 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2023-07-29T17:04:14.599 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-29T17:04:14.599 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2023-07-29T17:04:14.600 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-29T17:04:14.875 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2023-07-29T17:04:14.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-29T17:04:14.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-29T17:04:15.145 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2023-07-29T17:04:15.146 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2023-07-29T17:04:15.146 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:14 smithi129 bash[37985]: t=2023-07-29T17:04:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2023-07-29T17:04:15.146 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2023-07-29T17:04:15.146 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2023-07-29T17:04:15.409 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2023-07-29T17:04:15.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2023-07-29T17:04:15.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2023-07-29T17:04:15.410 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2023-07-29T17:04:15.797 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2023-07-29T17:04:15.797 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2023-07-29T17:04:15.798 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2023-07-29T17:04:15.798 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-29T17:04:16.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2023-07-29T17:04:16.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2023-07-29T17:04:16.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:15 smithi129 bash[37985]: t=2023-07-29T17:04:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2023-07-29T17:04:16.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2023-07-29T17:04:16.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2023-07-29T17:04:16.358 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2023-07-29T17:04:16.359 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2023-07-29T17:04:16.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[23526]: cluster 2023-07-29T17:04:15.123861+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:16.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:16 smithi078 bash[28259]: cluster 2023-07-29T17:04:15.123861+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:16.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:16 smithi078 bash[23704]: cluster 2023-07-29T17:04:15.123861+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:16.652 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2023-07-29T17:04:16.652 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2023-07-29T17:04:16.652 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2023-07-29T17:04:16.652 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2023-07-29T17:04:16.653 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2023-07-29T17:04:16.653 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2023-07-29T17:04:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2023-07-29T17:04:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2023-07-29T17:04:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2023-07-29T17:04:17.264 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:16 smithi129 bash[37985]: t=2023-07-29T17:04:16+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2023-07-29T17:04:17.264 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2023-07-29T17:04:17.264 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2023-07-29T17:04:17.264 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2023-07-29T17:04:17.515 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2023-07-29T17:04:17.515 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2023-07-29T17:04:17.515 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2023-07-29T17:04:17.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2023-07-29T17:04:17.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2023-07-29T17:04:17.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2023-07-29T17:04:17.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2023-07-29T17:04:18.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2023-07-29T17:04:18.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-29T17:04:18.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-29T17:04:18.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:17 smithi129 bash[37985]: t=2023-07-29T17:04:17+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2023-07-29T17:04:18.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[23526]: cluster 2023-07-29T17:04:17.124411+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:18.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2023-07-29T17:04:18.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2023-07-29T17:04:18.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2023-07-29T17:04:18.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-29T17:04:18.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:18 smithi078 bash[28259]: cluster 2023-07-29T17:04:17.124411+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:18.436 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:18 smithi078 bash[23704]: cluster 2023-07-29T17:04:17.124411+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:18.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2023-07-29T17:04:18.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2023-07-29T17:04:18.661 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-29T17:04:18.661 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2023-07-29T17:04:18.661 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2023-07-29T17:04:18.949 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2023-07-29T17:04:18.949 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2023-07-29T17:04:18.949 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2023-07-29T17:04:18.949 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2023-07-29T17:04:18.950 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:18 smithi129 bash[37985]: t=2023-07-29T17:04:18+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2023-07-29T17:04:19.241 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2023-07-29T17:04:19.242 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2023-07-29T17:04:19.242 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2023-07-29T17:04:19.527 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2023-07-29T17:04:19.527 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2023-07-29T17:04:19.527 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2023-07-29T17:04:19.528 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2023-07-29T17:04:19.528 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2023-07-29T17:04:19.836 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2023-07-29T17:04:19.837 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2023-07-29T17:04:19.837 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2023-07-29T17:04:20.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2023-07-29T17:04:20.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2023-07-29T17:04:20.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:19 smithi129 bash[37985]: t=2023-07-29T17:04:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-29T17:04:20.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2023-07-29T17:04:20.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2023-07-29T17:04:20.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[23526]: cluster 2023-07-29T17:04:19.125092+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:20.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2023-07-29T17:04:20.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2023-07-29T17:04:20.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2023-07-29T17:04:20.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2023-07-29T17:04:20.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2023-07-29T17:04:20.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2023-07-29T17:04:20.660 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2023-07-29T17:04:20.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:20 smithi078 bash[28259]: cluster 2023-07-29T17:04:19.125092+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:20.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:20 smithi078 bash[23704]: cluster 2023-07-29T17:04:19.125092+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:20.960 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-29T17:04:20.960 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2023-07-29T17:04:20.960 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2023-07-29T17:04:20.961 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-29T17:04:20.961 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:20 smithi129 bash[37985]: t=2023-07-29T17:04:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2023-07-29T17:04:21.228 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2023-07-29T17:04:21.228 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2023-07-29T17:04:21.479 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2023-07-29T17:04:21.479 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2023-07-29T17:04:21.480 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2023-07-29T17:04:21.480 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2023-07-29T17:04:21.480 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2023-07-29T17:04:21.798 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2023-07-29T17:04:21.798 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2023-07-29T17:04:21.798 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2023-07-29T17:04:21.798 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2023-07-29T17:04:22.049 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2023-07-29T17:04:22.050 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2023-07-29T17:04:22.050 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:21 smithi129 bash[37985]: t=2023-07-29T17:04:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2023-07-29T17:04:22.050 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2023-07-29T17:04:22.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[23526]: cluster 2023-07-29T17:04:21.125736+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:22.360 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2023-07-29T17:04:22.360 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-29T17:04:22.360 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2023-07-29T17:04:22.619 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2023-07-29T17:04:22.619 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2023-07-29T17:04:22.620 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2023-07-29T17:04:22.620 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2023-07-29T17:04:22.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:22 smithi078 bash[23704]: cluster 2023-07-29T17:04:21.125736+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:22.662 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:22 smithi078 bash[28259]: cluster 2023-07-29T17:04:21.125736+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:22.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2023-07-29T17:04:22.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2023-07-29T17:04:22.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2023-07-29T17:04:22.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2023-07-29T17:04:23.197 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2023-07-29T17:04:23.198 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:22 smithi129 bash[37985]: t=2023-07-29T17:04:22+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2023-07-29T17:04:23.198 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-29T17:04:23.198 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-29T17:04:23.198 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-29T17:04:23.507 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-29T17:04:23.508 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2023-07-29T17:04:23.508 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2023-07-29T17:04:23.801 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2023-07-29T17:04:23.801 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2023-07-29T17:04:23.801 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-29T17:04:23.802 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2023-07-29T17:04:23.802 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2023-07-29T17:04:24.103 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2023-07-29T17:04:24.103 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:23 smithi129 bash[37985]: t=2023-07-29T17:04:23+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2023-07-29T17:04:24.103 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2023-07-29T17:04:24.354 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[23526]: cluster 2023-07-29T17:04:23.126416+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:24.355 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2023-07-29T17:04:24.355 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2023-07-29T17:04:24.356 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2023-07-29T17:04:24.356 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2023-07-29T17:04:24.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:24 smithi078 bash[23704]: cluster 2023-07-29T17:04:23.126416+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:24.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:24 smithi078 bash[28259]: cluster 2023-07-29T17:04:23.126416+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:24.619 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2023-07-29T17:04:24.620 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2023-07-29T17:04:24.620 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2023-07-29T17:04:24.620 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2023-07-29T17:04:24.907 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2023-07-29T17:04:24.908 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-29T17:04:24.908 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-29T17:04:24.908 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2023-07-29T17:04:25.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2023-07-29T17:04:25.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:24 smithi129 bash[37985]: t=2023-07-29T17:04:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2023-07-29T17:04:25.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2023-07-29T17:04:25.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2023-07-29T17:04:25.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2023-07-29T17:04:25.528 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-29T17:04:25.528 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2023-07-29T17:04:25.528 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2023-07-29T17:04:25.528 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2023-07-29T17:04:25.529 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2023-07-29T17:04:25.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2023-07-29T17:04:25.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-29T17:04:25.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2023-07-29T17:04:26.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2023-07-29T17:04:26.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2023-07-29T17:04:26.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:25 smithi129 bash[37985]: t=2023-07-29T17:04:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2023-07-29T17:04:26.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2023-07-29T17:04:26.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2023-07-29T17:04:26.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:26 smithi078 bash[23704]: cluster 2023-07-29T17:04:25.127061+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:26.343 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:26 smithi078 bash[28259]: cluster 2023-07-29T17:04:25.127061+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:26.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[23526]: cluster 2023-07-29T17:04:25.127061+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:26.441 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2023-07-29T17:04:26.442 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2023-07-29T17:04:26.442 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2023-07-29T17:04:26.442 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2023-07-29T17:04:26.709 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2023-07-29T17:04:26.710 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2023-07-29T17:04:26.710 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2023-07-29T17:04:27.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2023-07-29T17:04:27.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing HTTPServer" logger=server 2023-07-29T17:04:27.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2023-07-29T17:04:27.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing PluginManager" logger=server 2023-07-29T17:04:27.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Starting plugin search" logger=plugins 2023-07-29T17:04:27.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2023-07-29T17:04:27.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing HooksService" logger=server 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing RemoteCache" logger=server 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing RenderingService" logger=server 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing AlertEngine" logger=server 2023-07-29T17:04:27.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing QuotaService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing ServerLockService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing LoginService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing SearchService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing TracingService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2023-07-29T17:04:27.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing CleanUpService" logger=server 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing NotificationService" logger=server 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2023-07-29T17:04:27.164 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="Initializing Stream Manager" 2023-07-29T17:04:27.165 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:04:26 smithi129 bash[37985]: t=2023-07-29T17:04:26+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2023-07-29T17:04:27.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:27 smithi129 bash[23526]: cluster 2023-07-29T17:04:27.127742+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:27.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:27 smithi078 bash[23704]: cluster 2023-07-29T17:04:27.127742+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:27.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:27 smithi078 bash[28259]: cluster 2023-07-29T17:04:27.127742+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:28.241 INFO:teuthology.orchestra.run.smithi078.stdout:133143986210 2023-07-29T17:04:28.242 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-29T17:04:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:34359738423 2023-07-29T17:04:28.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-29T17:04:30.209 INFO:teuthology.orchestra.run.smithi078.stdout:111669149736 2023-07-29T17:04:30.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-29T17:04:30.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:30 smithi078 bash[28259]: cluster 2023-07-29T17:04:29.128269+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:30.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:30 smithi078 bash[23704]: cluster 2023-07-29T17:04:29.128269+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:30.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:30 smithi129 bash[23526]: cluster 2023-07-29T17:04:29.128269+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:32.103 INFO:teuthology.orchestra.run.smithi078.stdout:210453397525 2023-07-29T17:04:32.103 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-29T17:04:32.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:32 smithi078 bash[23704]: cluster 2023-07-29T17:04:31.128954+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:32.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:32 smithi078 bash[28259]: cluster 2023-07-29T17:04:31.128954+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:32.351 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:32 smithi078 systemd[1]: Stopping Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:04:32.352 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:32 smithi078 bash[41885]: level=info ts=2023-07-29T17:04:32.345Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:04:32.419 INFO:teuthology.orchestra.run.smithi078.stdout:77309411373 2023-07-29T17:04:32.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-29T17:04:32.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:32 smithi129 bash[23526]: cluster 2023-07-29T17:04:31.128954+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:33.950 INFO:teuthology.orchestra.run.smithi078.stdout:184683593754 2023-07-29T17:04:33.951 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-29T17:04:34.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:34 smithi078 bash[23704]: cluster 2023-07-29T17:04:33.129552+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:34.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:34 smithi078 bash[28259]: cluster 2023-07-29T17:04:33.129552+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:34.409 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:34 smithi078 docker[44524]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-alertmanager.a 2023-07-29T17:04:34.468 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:34 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-29T17:04:34.468 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:34 smithi078 systemd[1]: Stopped Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:04:34.469 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:34 smithi078 systemd[1]: Started Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:04:34.580 INFO:teuthology.orchestra.run.smithi078.stdout:158913789983 2023-07-29T17:04:34.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-29T17:04:34.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:34 smithi129 bash[23526]: cluster 2023-07-29T17:04:33.129552+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:35.105 INFO:teuthology.orchestra.run.smithi078.stdout:55834574899 2023-07-29T17:04:35.105 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-29T17:04:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:35.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[28259]: audit 2023-07-29T17:04:34.445720+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:35.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[28259]: cephadm 2023-07-29T17:04:34.447638+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:04:35.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[28259]: cephadm 2023-07-29T17:04:34.452918+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:04:35.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[28259]: cluster 2023-07-29T17:04:35.130166+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:35.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[28259]: audit 2023-07-29T17:04:35.251442+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:04:35.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[28259]: audit 2023-07-29T17:04:35.254954+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:04:35.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[23704]: audit 2023-07-29T17:04:34.445720+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:35.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[23704]: cephadm 2023-07-29T17:04:34.447638+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:04:35.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[23704]: cephadm 2023-07-29T17:04:34.452918+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:04:35.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[23704]: cluster 2023-07-29T17:04:35.130166+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:35.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[23704]: audit 2023-07-29T17:04:35.251442+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:04:35.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:35 smithi078 bash[23704]: audit 2023-07-29T17:04:35.254954+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:04:35.730 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:35 smithi129 bash[23526]: audit 2023-07-29T17:04:34.445720+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:35.730 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:35 smithi129 bash[23526]: cephadm 2023-07-29T17:04:34.447638+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:04:35.730 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:35 smithi129 bash[23526]: cephadm 2023-07-29T17:04:34.452918+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:04:35.730 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:35 smithi129 bash[23526]: cluster 2023-07-29T17:04:35.130166+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:35.731 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:35 smithi129 bash[23526]: audit 2023-07-29T17:04:35.251442+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:04:35.731 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:35 smithi129 bash[23526]: audit 2023-07-29T17:04:35.254954+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:04:36.058 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:37.097 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:38.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[23526]: cluster 2023-07-29T17:04:37.130562+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:38.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[23704]: cluster 2023-07-29T17:04:37.130562+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:38.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[28259]: cluster 2023-07-29T17:04:37.130562+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:38.427 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.198Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2023-07-29T17:04:38.428 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.199Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2023-07-29T17:04:38.428 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.201Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.78 port=9094 2023-07-29T17:04:38.428 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.202Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-29T17:04:38.428 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.250Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:04:38.428 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.250Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:04:38.428 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:38 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:38.255Z caller=main.go:497 msg=Listening address=:9093 2023-07-29T17:04:38.911 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 systemd[1]: Stopping Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:04:38.911 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=warn ts=2023-07-29T17:04:38.502Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:04:38.911 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:04:38.911 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:04:38.911 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.502Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.503Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:04:38.912 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.503Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:04:38.913 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:38 smithi129 bash[36890]: level=info ts=2023-07-29T17:04:38.503Z caller=main.go:761 msg="See you next time!" 2023-07-29T17:04:39.410 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:39 smithi129 docker[38587]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-prometheus.a 2023-07-29T17:04:39.411 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:39 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Succeeded. 2023-07-29T17:04:39.411 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:39 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:04:39.411 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:39 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:04:40.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:40.203Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000156089s 2023-07-29T17:04:40.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: cluster 2023-07-29T17:04:39.131209+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:40.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.230207+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.234855+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:04:40.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.237900+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078:9093"}]: dispatch 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.247623+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.257095+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.268227+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.277471+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.285247+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.294865+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:04:40.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.301693+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[23704]: audit 2023-07-29T17:04:39.304966+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:40.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: cluster 2023-07-29T17:04:39.131209+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:40.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.230207+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.234855+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:04:40.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.237900+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078:9093"}]: dispatch 2023-07-29T17:04:40.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.247623+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.257095+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:04:40.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.268227+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:04:40.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.277471+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.285247+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:04:40.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.294865+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:04:40.317 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.301693+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.317 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:40 smithi078 bash[28259]: audit 2023-07-29T17:04:39.304966+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:40.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: cluster 2023-07-29T17:04:39.131209+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:40.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.230207+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.234855+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:04:40.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.237900+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078:9093"}]: dispatch 2023-07-29T17:04:40.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.247623+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.257095+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:04:40.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.268227+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:04:40.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.277471+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.285247+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:04:40.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.294865+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:04:40.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.301693+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:40.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[23526]: audit 2023-07-29T17:04:39.304966+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:40.413 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.300Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:04:40.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.300Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:04:40.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.300Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:04:40.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.300Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:04:40.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.300Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:04:40.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.300Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:04:40.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.302Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:04:40.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.302Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:04:40.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.307Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:04:40.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.307Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2023-07-29T17:04:40.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2023-07-29T17:04:40.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.308Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.288932ms 2023-07-29T17:04:40.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.310Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:04:40.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.310Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:04:40.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.310Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:04:40.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.312Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:04:40.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:04:40 smithi129 bash[38636]: level=info ts=2023-07-29T17:04:40.312Z caller=main.go:646 msg="Server is ready to receive web requests." 2023-07-29T17:04:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[23704]: audit 2023-07-29T17:04:39.235429+0000 mgr.y (mgr.14144) 246 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:04:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[23704]: audit 2023-07-29T17:04:39.238649+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078:9093"}]: dispatch 2023-07-29T17:04:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[23704]: audit 2023-07-29T17:04:39.257756+0000 mgr.y (mgr.14144) 248 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:04:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[23704]: audit 2023-07-29T17:04:39.268954+0000 mgr.y (mgr.14144) 249 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:04:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[23704]: audit 2023-07-29T17:04:39.286101+0000 mgr.y (mgr.14144) 250 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:04:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[23704]: audit 2023-07-29T17:04:39.295583+0000 mgr.y (mgr.14144) 251 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:04:41.341 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[28259]: audit 2023-07-29T17:04:39.235429+0000 mgr.y (mgr.14144) 246 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:04:41.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[28259]: audit 2023-07-29T17:04:39.238649+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078:9093"}]: dispatch 2023-07-29T17:04:41.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[28259]: audit 2023-07-29T17:04:39.257756+0000 mgr.y (mgr.14144) 248 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:04:41.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[28259]: audit 2023-07-29T17:04:39.268954+0000 mgr.y (mgr.14144) 249 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:04:41.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[28259]: audit 2023-07-29T17:04:39.286101+0000 mgr.y (mgr.14144) 250 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:04:41.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:41 smithi078 bash[28259]: audit 2023-07-29T17:04:39.295583+0000 mgr.y (mgr.14144) 251 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:04:41.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:41 smithi129 bash[23526]: audit 2023-07-29T17:04:39.235429+0000 mgr.y (mgr.14144) 246 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:04:41.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:41 smithi129 bash[23526]: audit 2023-07-29T17:04:39.238649+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078:9093"}]: dispatch 2023-07-29T17:04:41.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:41 smithi129 bash[23526]: audit 2023-07-29T17:04:39.257756+0000 mgr.y (mgr.14144) 248 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:04:41.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:41 smithi129 bash[23526]: audit 2023-07-29T17:04:39.268954+0000 mgr.y (mgr.14144) 249 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:04:41.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:41 smithi129 bash[23526]: audit 2023-07-29T17:04:39.286101+0000 mgr.y (mgr.14144) 250 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:04:41.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:41 smithi129 bash[23526]: audit 2023-07-29T17:04:39.295583+0000 mgr.y (mgr.14144) 251 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:04:42.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:42 smithi078 bash[28259]: cluster 2023-07-29T17:04:41.131812+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:42.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:42 smithi078 bash[23704]: cluster 2023-07-29T17:04:41.131812+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:42.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:42 smithi129 bash[23526]: cluster 2023-07-29T17:04:41.131812+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:44.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:44 smithi078 bash[23704]: cluster 2023-07-29T17:04:43.132371+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:44.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:44 smithi078 bash[23704]: audit 2023-07-29T17:04:43.180636+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:44.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:44 smithi078 bash[28259]: cluster 2023-07-29T17:04:43.132371+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:44.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:44 smithi078 bash[28259]: audit 2023-07-29T17:04:43.180636+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:44.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:44 smithi129 bash[23526]: cluster 2023-07-29T17:04:43.132371+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:44.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:44 smithi129 bash[23526]: audit 2023-07-29T17:04:43.180636+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:46.412 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:46 smithi078 bash[28259]: cluster 2023-07-29T17:04:45.132995+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:46.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:46 smithi078 bash[23704]: cluster 2023-07-29T17:04:45.132995+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:46.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:46 smithi129 bash[23526]: cluster 2023-07-29T17:04:45.132995+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:48.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:48 smithi078 bash[23704]: cluster 2023-07-29T17:04:47.133572+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:48.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:48 smithi078 bash[28259]: cluster 2023-07-29T17:04:47.133572+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:48.449 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:04:48 smithi078 bash[44801]: level=info ts=2023-07-29T17:04:48.203Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000866764s 2023-07-29T17:04:48.534 INFO:teuthology.orchestra.run.smithi078.stdout:210453397528 2023-07-29T17:04:48.642 INFO:teuthology.orchestra.run.smithi078.stdout:111669149740 2023-07-29T17:04:48.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:48 smithi129 bash[23526]: cluster 2023-07-29T17:04:47.133572+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:48.975 INFO:teuthology.orchestra.run.smithi078.stdout:158913789986 2023-07-29T17:04:49.501 INFO:teuthology.orchestra.run.smithi078.stdout:34359738428 2023-07-29T17:04:49.580 INFO:teuthology.orchestra.run.smithi078.stdout:77309411376 2023-07-29T17:04:49.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:48.505389+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:49.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:48.527618+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? 172.21.15.78:0/3700077260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-29T17:04:49.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:48.636909+0000 mon.a (mon.0) 518 : audit [DBG] from='client.? 172.21.15.78:0/445239832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-29T17:04:49.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:48.746103+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:49.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:48.749524+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:49.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:48.969201+0000 mon.a (mon.0) 521 : audit [DBG] from='client.? 172.21.15.78:0/3001308211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-29T17:04:49.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: cluster 2023-07-29T17:04:49.135038+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:49.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[28259]: audit 2023-07-29T17:04:49.494829+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.21.15.78:0/2338068158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-29T17:04:49.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:48.505389+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:49.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:48.527618+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? 172.21.15.78:0/3700077260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-29T17:04:49.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:48.636909+0000 mon.a (mon.0) 518 : audit [DBG] from='client.? 172.21.15.78:0/445239832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-29T17:04:49.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:48.746103+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:49.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:48.749524+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:49.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:48.969201+0000 mon.a (mon.0) 521 : audit [DBG] from='client.? 172.21.15.78:0/3001308211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-29T17:04:49.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: cluster 2023-07-29T17:04:49.135038+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:49.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:49 smithi078 bash[23704]: audit 2023-07-29T17:04:49.494829+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.21.15.78:0/2338068158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-29T17:04:49.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:48.505389+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:49.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:48.527618+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? 172.21.15.78:0/3700077260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-29T17:04:49.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:48.636909+0000 mon.a (mon.0) 518 : audit [DBG] from='client.? 172.21.15.78:0/445239832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-29T17:04:49.912 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:48.746103+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:49.912 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:48.749524+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:04:49.912 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:48.969201+0000 mon.a (mon.0) 521 : audit [DBG] from='client.? 172.21.15.78:0/3001308211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-29T17:04:49.912 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: cluster 2023-07-29T17:04:49.135038+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:49.912 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:49 smithi129 bash[23526]: audit 2023-07-29T17:04:49.494829+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.21.15.78:0/2338068158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-29T17:04:50.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:50 smithi078 bash[23704]: audit 2023-07-29T17:04:49.574034+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.21.15.78:0/2182389779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-29T17:04:50.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:50 smithi078 bash[28259]: audit 2023-07-29T17:04:49.574034+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.21.15.78:0/2182389779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-29T17:04:50.788 INFO:teuthology.orchestra.run.smithi078.stdout:133143986215 2023-07-29T17:04:50.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:50 smithi129 bash[23526]: audit 2023-07-29T17:04:49.574034+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.21.15.78:0/2182389779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-29T17:04:51.105 INFO:teuthology.orchestra.run.smithi078.stdout:55834574902 2023-07-29T17:04:51.224 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149736 got 111669149740 for osd.3 2023-07-29T17:04:51.224 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:51.569 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397525 got 210453397528 for osd.7 2023-07-29T17:04:51.570 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:51.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:51 smithi129 bash[23526]: audit 2023-07-29T17:04:50.782104+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.21.15.78:0/3260290021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-29T17:04:51.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:51 smithi129 bash[23526]: audit 2023-07-29T17:04:51.099362+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.21.15.78:0/1275752493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-29T17:04:51.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:51 smithi129 bash[23526]: cluster 2023-07-29T17:04:51.136149+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:51.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:51 smithi078 bash[28259]: audit 2023-07-29T17:04:50.782104+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.21.15.78:0/3260290021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-29T17:04:51.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:51 smithi078 bash[28259]: audit 2023-07-29T17:04:51.099362+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.21.15.78:0/1275752493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-29T17:04:51.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:51 smithi078 bash[28259]: cluster 2023-07-29T17:04:51.136149+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:51.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:51 smithi078 bash[23704]: audit 2023-07-29T17:04:50.782104+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.21.15.78:0/3260290021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-29T17:04:51.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:51 smithi078 bash[23704]: audit 2023-07-29T17:04:51.099362+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.21.15.78:0/1275752493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-29T17:04:51.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:51 smithi078 bash[23704]: cluster 2023-07-29T17:04:51.136149+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:52.324 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789983 got 158913789986 for osd.5 2023-07-29T17:04:52.324 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:52.667 INFO:teuthology.orchestra.run.smithi078.stdout:184683593757 2023-07-29T17:04:53.176 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738423 got 34359738428 for osd.0 2023-07-29T17:04:53.177 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:53.230 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411373 got 77309411376 for osd.2 2023-07-29T17:04:53.231 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:53.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574899 got 55834574902 for osd.1 2023-07-29T17:04:53.778 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:53.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986210 got 133143986215 for osd.4 2023-07-29T17:04:53.837 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:53.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:53 smithi129 bash[23526]: audit 2023-07-29T17:04:52.643319+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:53.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:53 smithi129 bash[23526]: audit 2023-07-29T17:04:52.661263+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.78:0/207033828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-29T17:04:53.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:53 smithi129 bash[23526]: cluster 2023-07-29T17:04:53.136773+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:53.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:53 smithi078 bash[28259]: audit 2023-07-29T17:04:52.643319+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:53.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:53 smithi078 bash[28259]: audit 2023-07-29T17:04:52.661263+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.78:0/207033828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-29T17:04:53.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:53 smithi078 bash[28259]: cluster 2023-07-29T17:04:53.136773+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:53.913 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:53 smithi078 bash[23704]: audit 2023-07-29T17:04:52.643319+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:53.913 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:53 smithi078 bash[23704]: audit 2023-07-29T17:04:52.661263+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.78:0/207033828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-29T17:04:53.913 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:53 smithi078 bash[23704]: cluster 2023-07-29T17:04:53.136773+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:53.914 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593754 got 184683593757 for osd.6 2023-07-29T17:04:53.914 DEBUG:teuthology.parallel:result is None 2023-07-29T17:04:53.914 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-29T17:04:53.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph pg dump --format=json 2023-07-29T17:04:55.757 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:04:56.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:56 smithi129 bash[23526]: cluster 2023-07-29T17:04:55.137311+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:56.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:56 smithi129 bash[23526]: audit 2023-07-29T17:04:55.703723+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:56.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:56 smithi129 bash[23526]: audit 2023-07-29T17:04:55.711445+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:56.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:56 smithi078 bash[23704]: cluster 2023-07-29T17:04:55.137311+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:56.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:56 smithi078 bash[23704]: audit 2023-07-29T17:04:55.703723+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:56.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:56 smithi078 bash[23704]: audit 2023-07-29T17:04:55.711445+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:56.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:56 smithi078 bash[28259]: cluster 2023-07-29T17:04:55.137311+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:56.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:56 smithi078 bash[28259]: audit 2023-07-29T17:04:55.703723+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:56.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:56 smithi078 bash[28259]: audit 2023-07-29T17:04:55.711445+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:04:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:04:58 smithi078 bash[23704]: cluster 2023-07-29T17:04:57.137955+0000 mgr.y (mgr.14144) 260 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:58.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:04:58 smithi078 bash[28259]: cluster 2023-07-29T17:04:57.137955+0000 mgr.y (mgr.14144) 260 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:58.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:04:58 smithi129 bash[23526]: cluster 2023-07-29T17:04:57.137955+0000 mgr.y (mgr.14144) 260 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:04:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:04:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-29T17:04:59.577 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2023-07-29T17:04:57.137608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43284,"kb_used_data":3796,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":749917932,"statfs":{"total":767960285184,"available":767915962368,"internally_reserved":0,"allocated":3887104,"data_stored":1532439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"51","state":"active+clean","last_fresh":"2023-07-29T17:03:01.740647+0000","last_change":"2023-07-29T17:03:01.740647+0000","last_active":"2023-07-29T17:03:01.740647+0000","last_peered":"2023-07-29T17:03:01.740647+0000","last_clean":"2023-07-29T17:03:01.740647+0000","last_became_active":"2023-07-29T17:03:01.739884+0000","last_became_peered":"2023-07-29T17:03:01.739884+0000","last_unstale":"2023-07-29T17:03:01.740647+0000","last_undegraded":"2023-07-29T17:03:01.740647+0000","last_fullsized":"2023-07-29T17:03:01.740647+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T17:01:12.204854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T17:01:12.204854+0000","last_clean_scrub_stamp":"2023-07-29T17:01:12.204854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5220,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739932,"statfs":{"total":95995035648,"available":95989690368,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.69699999999999995}]},{"osd":1,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.0840000000000001}]},{"osd":4,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.997},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":1.0269999999999999}]},{"osd":6,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.056}]}]},{"osd":6,"up_from":43,"seq":184683593759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5280,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.96699999999999997,"15min":0.96899999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.357,"5min":1.71,"15min":1.71},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.91400000000000003,"15min":0.90200000000000002},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.046}]},{"osd":1,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92600000000000005,"15min":0.93000000000000005},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.2609999999999999,"5min":1.496,"15min":1.496},"last":1.171},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.0189999999999999},"min":{"1min":0.70699999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.2450000000000001,"5min":1.534,"15min":1.534},"last":0.78900000000000003}]},{"osd":2,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0009999999999999,"15min":1.0089999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.2070000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.006,"5min":0.95899999999999996,"15min":0.95099999999999996},"min":{"1min":0.69299999999999995,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.83799999999999997}]},{"osd":3,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.97399999999999998,"15min":0.96999999999999997},"min":{"1min":0.75600000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.872},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.129,"15min":1.125},"min":{"1min":0.85999999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.504,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.2050000000000001}]},{"osd":4,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.3440000000000001,"5min":1.375,"15min":1.375},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.0109999999999999,"15min":1.01},"min":{"1min":0.70399999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.99199999999999999}]},{"osd":5,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.052,"15min":1.04},"min":{"1min":0.72599999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.0880000000000001,"15min":1.0840000000000001},"min":{"1min":0.65000000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.093}]},{"osd":7,"last update":"Sat Jul 29 17:04:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.149}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5536,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739616,"statfs":{"total":95995035648,"available":95989366784,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.024,"15min":1.3560000000000001},"min":{"1min":0.47599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1990000000000001,"5min":21.248000000000001,"15min":21.248000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.1180000000000001,"15min":1.466},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0700000000000001,"5min":21.838000000000001,"15min":21.838000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Sat Jul 29 17:03:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.749},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0069999999999999,"5min":1.53,"15min":1.53},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.83099999999999996,"15min":0.81999999999999995},"min":{"1min":0.66500000000000004,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1879999999999999,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":0.94999999999999996}]},{"osd":3,"last update":"Sat Jul 29 17:04:27 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.84899999999999998,"15min":0.81599999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.81799999999999995,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.22,"5min":1.298,"15min":1.298},"last":0.71199999999999997}]},{"osd":4,"last update":"Sat Jul 29 17:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.052,"15min":1.0609999999999999},"min":{"1min":0.79300000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.1180000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.181},{"interface":"front","average":{"1min":1.028,"5min":1.022,"15min":1.03},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.1639999999999999}]},{"osd":5,"last update":"Sat Jul 29 17:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.115,"15min":1.135},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.353,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.103},{"interface":"front","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.1830000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.331,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.375}]},{"osd":6,"last update":"Sat Jul 29 17:04:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.2250000000000001,"15min":1.2430000000000001},"min":{"1min":0.91500000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.3240000000000001,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1799999999999999,"15min":1.1890000000000001},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.484,"5min":1.571,"15min":1.571},"last":1.04}]},{"osd":7,"last update":"Sat Jul 29 17:04:00 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":1.2869999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5604,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739548,"statfs":{"total":95995035648,"available":95989297152,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 17:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96299999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.626},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.749,"15min":0.65900000000000003},"min":{"1min":0.497,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Jul 29 17:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.71299999999999997,"15min":0.68600000000000005},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.78000000000000003,"15min":0.748},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.79500000000000004}]},{"osd":3,"last update":"Sat Jul 29 17:04:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78700000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.873,"15min":0.872},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.1180000000000001,"5min":1.236,"15min":1.236},"last":0.66000000000000003}]},{"osd":4,"last update":"Sat Jul 29 17:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.0349999999999999,"15min":1.0309999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3089999999999999,"5min":1.546,"15min":1.546},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0840000000000001,"15min":1.0860000000000001},"min":{"1min":0.78400000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.476,"5min":1.631,"15min":1.631},"last":0.78400000000000003}]},{"osd":5,"last update":"Sat Jul 29 17:04:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.147,"15min":1.141},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.2050000000000001,"15min":1.2270000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.4350000000000001,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":3.1139999999999999}]},{"osd":6,"last update":"Sat Jul 29 17:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1839999999999999,"15min":1.194},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.677,"15min":1.677},"last":1.004},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.258,"15min":1.2769999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.3700000000000001,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.1299999999999999}]},{"osd":7,"last update":"Sat Jul 29 17:04:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":3.1859999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.9079999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5476,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739676,"statfs":{"total":95995035648,"available":95989428224,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:00 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.65800000000000003,"15min":0.63400000000000001},"min":{"1min":0.51500000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.59199999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95199999999999996,"5min":1.3,"15min":1.3},"last":0.95599999999999996}]},{"osd":1,"last update":"Sat Jul 29 17:04:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.51000000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.73299999999999998,"15min":0.71499999999999997},"min":{"1min":0.63800000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.85399999999999998}]},{"osd":3,"last update":"Sat Jul 29 17:04:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.83399999999999996,"15min":0.82199999999999995},"min":{"1min":0.55100000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.355,"5min":1.381,"15min":1.381},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.92400000000000004,"15min":0.90200000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":1.319}]},{"osd":4,"last update":"Sat Jul 29 17:03:49 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.96799999999999997,"15min":0.96599999999999997},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.071},{"interface":"front","average":{"1min":1.117,"5min":1.0369999999999999,"15min":1.0229999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.4790000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.92800000000000005}]},{"osd":5,"last update":"Sat Jul 29 17:04:15 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.075,"15min":1.069},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.034},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.111,"15min":1.1060000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.139}]},{"osd":6,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1719999999999999,"15min":1.1879999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.421,"5min":1.946,"15min":1.946},"last":1.123},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.22,"15min":1.226},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.212}]},{"osd":7,"last update":"Sat Jul 29 17:04:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":1.099,"5min":1.099,"15min":1.099},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":1.1759999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5412,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739740,"statfs":{"total":95995035648,"available":95989493760,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.66400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88600000000000001,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.754,"15min":0.73999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0660000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.93200000000000005}]},{"osd":1,"last update":"Sat Jul 29 17:04:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.68300000000000005}]},{"osd":2,"last update":"Sat Jul 29 17:04:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.83299999999999996,"15min":0.81499999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.83299999999999996,"15min":0.81100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1539999999999999,"5min":1.25,"15min":1.25},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Jul 29 17:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.04,"15min":1.044},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.196,"5min":1.462,"15min":1.462},"last":0.998},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0209999999999999,"15min":1.0009999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.361,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.95799999999999996}]},{"osd":5,"last update":"Sat Jul 29 17:04:12 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0900000000000001,"15min":1.0920000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.464,"5min":1.643,"15min":1.643},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.1200000000000001,"15min":1.1100000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.329,"5min":1.482,"15min":1.482},"last":1.085}]},{"osd":6,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.486,"5min":1.526,"15min":1.526},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.212,"5min":1.228,"15min":1.2310000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.476,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.026}]},{"osd":7,"last update":"Sat Jul 29 17:04:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.0529999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5412,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739740,"statfs":{"total":95995035648,"available":95989493760,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88400000000000001,"15min":0.876},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2450000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2130000000000001,"5min":1.698,"15min":1.698},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.92200000000000004,"15min":0.89800000000000002},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.266,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.94799999999999995,"15min":0.95599999999999996},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2789999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.89100000000000001}]},{"osd":2,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96499999999999997,"15min":0.95499999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.2130000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98599999999999999,"15min":0.97599999999999998},"min":{"1min":0.71199999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.179,"5min":1.649,"15min":1.649},"last":0.85599999999999998}]},{"osd":3,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.999,"15min":0.999},"min":{"1min":0.76900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.228,"5min":1.496,"15min":1.496},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.018,"15min":1.022},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.77500000000000002}]},{"osd":5,"last update":"Sat Jul 29 17:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.91600000000000004,"15min":0.91300000000000003},"min":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0740000000000001,"15min":1.0760000000000001},"min":{"1min":0.63100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.327,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.1899999999999999}]},{"osd":6,"last update":"Sat Jul 29 17:04:37 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98399999999999999,"15min":0.98599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.363,"5min":1.462,"15min":1.462},"last":1.095},{"interface":"front","average":{"1min":1.012,"5min":1.0780000000000001,"15min":1.089},"min":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.298,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.226}]},{"osd":7,"last update":"Sat Jul 29 17:03:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.115}]}]},{"osd":5,"up_from":37,"seq":158913789988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5344,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739808,"statfs":{"total":95995035648,"available":95989563392,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.94799999999999995,"15min":0.95599999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.2749999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.83799999999999997,"15min":0.82599999999999996},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.83799999999999997}]},{"osd":1,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.91700000000000004,"15min":0.91900000000000004},"min":{"1min":0.72899999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93300000000000005,"15min":0.93200000000000005},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88300000000000001,"15min":0.873},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.746},{"interface":"front","average":{"1min":1.01,"5min":0.97599999999999998,"15min":0.96999999999999997},"min":{"1min":0.752,"5min":0.745,"15min":0.745},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":1.23}]},{"osd":3,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.78400000000000003,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.3149999999999999,"5min":1.696,"15min":1.696},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.04,"15min":1.034},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.464,"5min":1.573,"15min":1.573},"last":1.0569999999999999}]},{"osd":4,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.87,"15min":0.85999999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.94599999999999995,"15min":0.93799999999999994},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.278,"5min":1.796,"15min":1.796},"last":0.97099999999999997}]},{"osd":6,"last update":"Sat Jul 29 17:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0209999999999999,"15min":1.016},"min":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.3340000000000001,"5min":1.415,"15min":1.415},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.216,"15min":1.2070000000000001},"min":{"1min":0.96199999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.4910000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.3140000000000001}]},{"osd":7,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.272},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.163}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T17:04:59.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph pg dump --format=json 2023-07-29T17:05:00.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:00 smithi129 bash[23526]: audit 2023-07-29T17:04:58.896229+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T17:05:00.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:00 smithi129 bash[23526]: cluster 2023-07-29T17:04:59.138628+0000 mgr.y (mgr.14144) 262 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:00.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:00 smithi078 bash[28259]: audit 2023-07-29T17:04:58.896229+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T17:05:00.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:00 smithi078 bash[28259]: cluster 2023-07-29T17:04:59.138628+0000 mgr.y (mgr.14144) 262 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:00.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:00 smithi078 bash[23704]: audit 2023-07-29T17:04:58.896229+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T17:05:00.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:00 smithi078 bash[23704]: cluster 2023-07-29T17:04:59.138628+0000 mgr.y (mgr.14144) 262 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:05:02.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:02 smithi129 bash[23526]: cluster 2023-07-29T17:05:01.139254+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:02.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:02 smithi078 bash[28259]: cluster 2023-07-29T17:05:01.139254+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:02.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:02 smithi078 bash[23704]: cluster 2023-07-29T17:05:01.139254+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:04.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:04 smithi078 bash[23704]: cluster 2023-07-29T17:05:03.139863+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:04.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:04 smithi078 bash[28259]: cluster 2023-07-29T17:05:03.139863+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:04.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:04 smithi129 bash[23526]: cluster 2023-07-29T17:05:03.139863+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:05.294 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:05:05.298 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-29T17:05:05.955 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2023-07-29T17:05:05.140138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43284,"kb_used_data":3796,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":749917932,"statfs":{"total":767960285184,"available":767915962368,"internally_reserved":0,"allocated":3887104,"data_stored":1532439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"51","state":"active+clean","last_fresh":"2023-07-29T17:03:01.740647+0000","last_change":"2023-07-29T17:03:01.740647+0000","last_active":"2023-07-29T17:03:01.740647+0000","last_peered":"2023-07-29T17:03:01.740647+0000","last_clean":"2023-07-29T17:03:01.740647+0000","last_became_active":"2023-07-29T17:03:01.739884+0000","last_became_peered":"2023-07-29T17:03:01.739884+0000","last_unstale":"2023-07-29T17:03:01.740647+0000","last_undegraded":"2023-07-29T17:03:01.740647+0000","last_fullsized":"2023-07-29T17:03:01.740647+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T17:01:12.204854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T17:01:12.204854+0000","last_clean_scrub_stamp":"2023-07-29T17:01:12.204854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397532,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5220,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739932,"statfs":{"total":95995035648,"available":95989690368,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0720000000000001,"15min":1.0860000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":0.98099999999999998},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.2170000000000001,"5min":1.327,"15min":1.327},"last":0.98299999999999998}]},{"osd":1,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.98999999999999999,"15min":0.996},"min":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.234,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.004,"5min":0.96799999999999997,"15min":0.96199999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2669999999999999,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":1.0580000000000001}]},{"osd":2,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0169999999999999,"15min":1.018},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.137},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0660000000000001,"15min":1.0669999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.125}]},{"osd":3,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0129999999999999,"15min":1.024},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.252,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1459999999999999,"15min":1.1579999999999999},"min":{"1min":0.83599999999999997,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.95499999999999996}]},{"osd":4,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.014,"15min":1.02},"min":{"1min":0.58799999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":1.171},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.077,"15min":1.081},"min":{"1min":0.78800000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.4350000000000001,"5min":1.629,"15min":1.629},"last":1.018}]},{"osd":5,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.173,"15min":1.1830000000000001},"min":{"1min":0.752,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.7130000000000001,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.1850000000000001,"15min":1.177},"min":{"1min":0.94899999999999995,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5209999999999999,"5min":1.595,"15min":1.595},"last":1.234}]},{"osd":6,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.76600000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.417,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.26,"15min":1.2709999999999999},"min":{"1min":0.875,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.536,"5min":1.911,"15min":1.911},"last":1.2030000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5280,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.96699999999999997,"15min":0.96899999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.357,"5min":1.71,"15min":1.71},"last":1.012},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.91400000000000003,"15min":0.90200000000000002},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.0600000000000001}]},{"osd":1,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92600000000000005,"15min":0.93000000000000005},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.2609999999999999,"5min":1.496,"15min":1.496},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.0189999999999999},"min":{"1min":0.70699999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.2450000000000001,"5min":1.534,"15min":1.534},"last":0.88700000000000001}]},{"osd":2,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0009999999999999,"15min":1.0089999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.2070000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.006,"5min":0.95899999999999996,"15min":0.95099999999999996},"min":{"1min":0.69299999999999995,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.3360000000000001}]},{"osd":3,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.97399999999999998,"15min":0.96999999999999997},"min":{"1min":0.75600000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.129,"15min":1.125},"min":{"1min":0.85999999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.504,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.274}]},{"osd":4,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.3440000000000001,"5min":1.375,"15min":1.375},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.0109999999999999,"15min":1.01},"min":{"1min":0.70399999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.1100000000000001}]},{"osd":5,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.052,"15min":1.04},"min":{"1min":0.72599999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.0880000000000001,"15min":1.0840000000000001},"min":{"1min":0.65000000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.21}]},{"osd":7,"last update":"Sat Jul 29 17:04:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.665}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5536,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739616,"statfs":{"total":95995035648,"available":95989366784,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.024,"15min":1.3560000000000001},"min":{"1min":0.47599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1990000000000001,"5min":21.248000000000001,"15min":21.248000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.1180000000000001,"15min":1.466},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0700000000000001,"5min":21.838000000000001,"15min":21.838000000000001},"last":1.1850000000000001}]},{"osd":2,"last update":"Sat Jul 29 17:04:58 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.754,"15min":0.749},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.147,"5min":1.53,"15min":1.53},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.81200000000000006,"15min":0.81399999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94999999999999996,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":0.77700000000000002}]},{"osd":3,"last update":"Sat Jul 29 17:04:27 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.84899999999999998,"15min":0.81599999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.81799999999999995,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.22,"5min":1.298,"15min":1.298},"last":0.92200000000000004}]},{"osd":4,"last update":"Sat Jul 29 17:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.052,"15min":1.0609999999999999},"min":{"1min":0.79300000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.1180000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.022,"15min":1.03},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.0600000000000001}]},{"osd":5,"last update":"Sat Jul 29 17:04:17 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.115,"15min":1.135},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.353,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.1699999999999999,"15min":1.1830000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.331,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Sat Jul 29 17:04:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.2250000000000001,"15min":1.2430000000000001},"min":{"1min":0.91500000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.3240000000000001,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1799999999999999,"15min":1.1890000000000001},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.484,"5min":1.571,"15min":1.571},"last":1.1419999999999999}]},{"osd":7,"last update":"Sat Jul 29 17:04:00 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":1.4770000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5604,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739548,"statfs":{"total":95995035648,"available":95989297152,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 17:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96299999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.749,"15min":0.65900000000000003},"min":{"1min":0.497,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.745}]},{"osd":2,"last update":"Sat Jul 29 17:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.71299999999999997,"15min":0.68600000000000005},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.78000000000000003,"15min":0.748},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.78700000000000003}]},{"osd":3,"last update":"Sat Jul 29 17:04:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78700000000000003,"15min":0.78100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.873,"15min":0.872},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.1180000000000001,"5min":1.236,"15min":1.236},"last":0.68200000000000005}]},{"osd":4,"last update":"Sat Jul 29 17:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.0349999999999999,"15min":1.0309999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3089999999999999,"5min":1.546,"15min":1.546},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0840000000000001,"15min":1.0860000000000001},"min":{"1min":0.78400000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.476,"5min":1.631,"15min":1.631},"last":1.0369999999999999}]},{"osd":5,"last update":"Sat Jul 29 17:04:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.147,"15min":1.141},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.2050000000000001,"15min":1.2270000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.4350000000000001,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.1479999999999999}]},{"osd":6,"last update":"Sat Jul 29 17:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1839999999999999,"15min":1.194},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6639999999999999,"5min":1.677,"15min":1.677},"last":1.222},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.258,"15min":1.2769999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.3700000000000001,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.179}]},{"osd":7,"last update":"Sat Jul 29 17:04:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.995},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.1180000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5476,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739676,"statfs":{"total":95995035648,"available":95989428224,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.69499999999999995,"15min":0.64700000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0229999999999999,"5min":1.3,"15min":1.3},"last":0.80700000000000005}]},{"osd":1,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.75700000000000001,"15min":0.73299999999999998},"min":{"1min":0.45300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.1080000000000001,"5min":1.133,"15min":1.133},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.753,"15min":0.72199999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1379999999999999,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":1.1160000000000001}]},{"osd":3,"last update":"Sat Jul 29 17:04:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.83399999999999996,"15min":0.82199999999999995},"min":{"1min":0.55100000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.355,"5min":1.381,"15min":1.381},"last":1.179},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.92400000000000004,"15min":0.90200000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.89000000000000001}]},{"osd":4,"last update":"Sat Jul 29 17:04:53 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.228,"5min":1.403,"15min":1.403},"last":0.878},{"interface":"front","average":{"1min":0.996,"5min":1.032,"15min":1.022},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.3939999999999999,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.0309999999999999}]},{"osd":5,"last update":"Sat Jul 29 17:04:15 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.075,"15min":1.069},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.204},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.111,"15min":1.1060000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.0609999999999999}]},{"osd":6,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1719999999999999,"15min":1.1879999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.421,"5min":1.946,"15min":1.946},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.22,"15min":1.226},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.3999999999999999}]},{"osd":7,"last update":"Sat Jul 29 17:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2769999999999999,"15min":1.2929999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.5189999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.228,"15min":1.24},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.504,"5min":1.679,"15min":1.679},"last":1.0820000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5412,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739740,"statfs":{"total":95995035648,"available":95989493760,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.66400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88600000000000001,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.754,"15min":0.73999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0660000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.90400000000000003}]},{"osd":1,"last update":"Sat Jul 29 17:04:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Jul 29 17:04:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.83299999999999996,"15min":0.81499999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.83299999999999996,"15min":0.81100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1539999999999999,"5min":1.25,"15min":1.25},"last":0.98799999999999999}]},{"osd":4,"last update":"Sat Jul 29 17:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.04,"15min":1.044},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.196,"5min":1.462,"15min":1.462},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0209999999999999,"15min":1.0009999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.361,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.133}]},{"osd":5,"last update":"Sat Jul 29 17:04:12 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0900000000000001,"15min":1.0920000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.464,"5min":1.643,"15min":1.643},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.1200000000000001,"15min":1.1100000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.329,"5min":1.482,"15min":1.482},"last":1.238}]},{"osd":6,"last update":"Sat Jul 29 17:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.486,"5min":1.526,"15min":1.526},"last":1.21},{"interface":"front","average":{"1min":1.212,"5min":1.228,"15min":1.2310000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.476,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.3089999999999999}]},{"osd":7,"last update":"Sat Jul 29 17:04:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.2709999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5412,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739740,"statfs":{"total":95995035648,"available":95989493760,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88400000000000001,"15min":0.876},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2450000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2130000000000001,"5min":1.698,"15min":1.698},"last":0.74199999999999999}]},{"osd":1,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.92200000000000004,"15min":0.89800000000000002},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.266,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.94799999999999995,"15min":0.95599999999999996},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2789999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":1.1879999999999999}]},{"osd":2,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96499999999999997,"15min":0.95499999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.2130000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98599999999999999,"15min":0.97599999999999998},"min":{"1min":0.71199999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.179,"5min":1.649,"15min":1.649},"last":1.159}]},{"osd":3,"last update":"Sat Jul 29 17:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.999,"15min":0.999},"min":{"1min":0.76900000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.228,"5min":1.496,"15min":1.496},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.018,"15min":1.022},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":1.004}]},{"osd":5,"last update":"Sat Jul 29 17:04:15 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.91600000000000004,"15min":0.91300000000000003},"min":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0740000000000001,"15min":1.0760000000000001},"min":{"1min":0.63100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.327,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.25}]},{"osd":6,"last update":"Sat Jul 29 17:04:37 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98399999999999999,"15min":0.98599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.363,"5min":1.462,"15min":1.462},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.012,"5min":1.0780000000000001,"15min":1.089},"min":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.298,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.04}]},{"osd":7,"last update":"Sat Jul 29 17:03:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.216}]}]},{"osd":5,"up_from":37,"seq":158913789990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5344,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739808,"statfs":{"total":95995035648,"available":95989563392,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.94799999999999995,"15min":0.95599999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.2749999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.83799999999999997,"15min":0.82599999999999996},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.92400000000000004}]},{"osd":1,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.91700000000000004,"15min":0.91900000000000004},"min":{"1min":0.72899999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93300000000000005,"15min":0.93200000000000005},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":1.1180000000000001}]},{"osd":2,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88300000000000001,"15min":0.873},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.091},{"interface":"front","average":{"1min":1.01,"5min":0.97599999999999998,"15min":0.96999999999999997},"min":{"1min":0.752,"5min":0.745,"15min":0.745},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":1.2889999999999999}]},{"osd":3,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.78400000000000003,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.3149999999999999,"5min":1.696,"15min":1.696},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.077,"5min":1.04,"15min":1.034},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.464,"5min":1.573,"15min":1.573},"last":1.198}]},{"osd":4,"last update":"Sat Jul 29 17:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.87,"15min":0.85999999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.745},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.94599999999999995,"15min":0.93799999999999994},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.278,"5min":1.796,"15min":1.796},"last":0.998}]},{"osd":6,"last update":"Sat Jul 29 17:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0209999999999999,"15min":1.016},"min":{"1min":0.73099999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.3340000000000001,"5min":1.415,"15min":1.415},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.216,"15min":1.2070000000000001},"min":{"1min":0.96199999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.4910000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.242}]},{"osd":7,"last update":"Sat Jul 29 17:04:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0369999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T17:05:05.957 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-29T17:05:05.958 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-29T17:05:05.958 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-29T17:05:05.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- ceph health --format=json 2023-07-29T17:05:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:06 smithi129 bash[23526]: cluster 2023-07-29T17:05:05.140372+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:06.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:06 smithi078 bash[23704]: cluster 2023-07-29T17:05:05.140372+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:06.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:06 smithi078 bash[28259]: cluster 2023-07-29T17:05:05.140372+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:07.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:07 smithi129 bash[23526]: audit 2023-07-29T17:05:05.280075+0000 mgr.y (mgr.14144) 266 : audit [DBG] from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T17:05:07.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:07 smithi129 bash[23526]: cluster 2023-07-29T17:05:07.140966+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:07.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:07 smithi078 bash[28259]: audit 2023-07-29T17:05:05.280075+0000 mgr.y (mgr.14144) 266 : audit [DBG] from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T17:05:07.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:07 smithi078 bash[28259]: cluster 2023-07-29T17:05:07.140966+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:07.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:07 smithi078 bash[23704]: audit 2023-07-29T17:05:05.280075+0000 mgr.y (mgr.14144) 266 : audit [DBG] from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T17:05:07.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:07 smithi078 bash[23704]: cluster 2023-07-29T17:05:07.140966+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:08.705 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T17:05:10.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:10 smithi129 bash[23526]: cluster 2023-07-29T17:05:09.141555+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:10.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:10 smithi078 bash[23704]: cluster 2023-07-29T17:05:09.141555+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:10.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:10 smithi078 bash[28259]: cluster 2023-07-29T17:05:09.141555+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-29T17:05:12.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:12 smithi078 bash[28259]: cluster 2023-07-29T17:05:11.142161+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:12.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:12 smithi078 bash[28259]: audit 2023-07-29T17:05:11.823567+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.21.15.78:0/3163929544' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-29T17:05:12.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:12 smithi078 bash[23704]: cluster 2023-07-29T17:05:11.142161+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:12.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:12 smithi078 bash[23704]: audit 2023-07-29T17:05:11.823567+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.21.15.78:0/3163929544' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-29T17:05:12.493 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-29T17:05:12.494 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-29T17:05:12.494 INFO:tasks.cephadm:Setup complete, yielding 2023-07-29T17:05:12.494 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-29T17:05:12.505 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-29T17:05:12.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2023-07-29T17:05:12.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:12 smithi129 bash[23526]: cluster 2023-07-29T17:05:11.142161+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:12.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:12 smithi129 bash[23526]: audit 2023-07-29T17:05:11.823567+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.21.15.78:0/3163929544' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-29T17:05:14.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:14 smithi078 bash[28259]: cluster 2023-07-29T17:05:13.142733+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:14.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:14 smithi078 bash[23704]: cluster 2023-07-29T17:05:13.142733+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:14.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:14 smithi129 bash[23526]: cluster 2023-07-29T17:05:13.142733+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:15.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:15 smithi129 bash[23526]: cluster 2023-07-29T17:05:15.143313+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:15.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:15 smithi078 bash[23704]: cluster 2023-07-29T17:05:15.143313+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:15.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:15 smithi078 bash[28259]: cluster 2023-07-29T17:05:15.143313+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:16.491 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-29T17:05:16.500 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-29T17:05:16.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2023-07-29T17:05:17.161 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:16 smithi129 bash[23526]: audit 2023-07-29T17:05:15.822108+0000 mon.a (mon.0) 531 : audit [INF] from='client.? 172.21.15.78:0/3035158687' entity='client.admin' 2023-07-29T17:05:17.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:16 smithi129 bash[23526]: audit 2023-07-29T17:05:15.836195+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:17.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:16 smithi078 bash[23704]: audit 2023-07-29T17:05:15.822108+0000 mon.a (mon.0) 531 : audit [INF] from='client.? 172.21.15.78:0/3035158687' entity='client.admin' 2023-07-29T17:05:17.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:16 smithi078 bash[23704]: audit 2023-07-29T17:05:15.836195+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:17.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:16 smithi078 bash[28259]: audit 2023-07-29T17:05:15.822108+0000 mon.a (mon.0) 531 : audit [INF] from='client.? 172.21.15.78:0/3035158687' entity='client.admin' 2023-07-29T17:05:17.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:16 smithi078 bash[28259]: audit 2023-07-29T17:05:15.836195+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:18.162 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:17 smithi129 bash[23526]: cluster 2023-07-29T17:05:17.143925+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:18.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:17 smithi078 bash[23704]: cluster 2023-07-29T17:05:17.143925+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:18.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:17 smithi078 bash[28259]: cluster 2023-07-29T17:05:17.143925+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:20.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:20 smithi129 bash[23526]: cluster 2023-07-29T17:05:19.144494+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:20.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:20 smithi078 bash[28259]: cluster 2023-07-29T17:05:19.144494+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:20.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:20 smithi078 bash[23704]: cluster 2023-07-29T17:05:19.144494+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:21.262 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:05:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "6816e162-7db8-4b7f-9381-135762243806", 2023-07-29T17:05:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "r", 2023-07-29T17:05:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "current_period": "fad7ffdd-6aa4-4df6-a289-ec229af69973", 2023-07-29T17:05:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1 2023-07-29T17:05:21.263 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:05:21.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:21 smithi129 bash[23526]: cluster 2023-07-29T17:05:20.212081+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2023-07-29T17:05:21.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:21 smithi129 bash[23526]: audit 2023-07-29T17:05:20.220077+0000 mon.c (mon.2) 16 : audit [INF] from='client.? 172.21.15.78:0/1598175408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-29T17:05:21.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:21 smithi129 bash[23526]: audit 2023-07-29T17:05:20.220434+0000 mon.a (mon.0) 534 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-29T17:05:21.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:21 smithi129 bash[23526]: audit 2023-07-29T17:05:21.212677+0000 mon.a (mon.0) 535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-29T17:05:21.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:21 smithi129 bash[23526]: cluster 2023-07-29T17:05:21.212808+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2023-07-29T17:05:21.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[28259]: cluster 2023-07-29T17:05:20.212081+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2023-07-29T17:05:21.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[28259]: audit 2023-07-29T17:05:20.220077+0000 mon.c (mon.2) 16 : audit [INF] from='client.? 172.21.15.78:0/1598175408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-29T17:05:21.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[28259]: audit 2023-07-29T17:05:20.220434+0000 mon.a (mon.0) 534 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-29T17:05:21.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[28259]: audit 2023-07-29T17:05:21.212677+0000 mon.a (mon.0) 535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-29T17:05:21.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[28259]: cluster 2023-07-29T17:05:21.212808+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2023-07-29T17:05:21.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[23704]: cluster 2023-07-29T17:05:20.212081+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2023-07-29T17:05:21.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[23704]: audit 2023-07-29T17:05:20.220077+0000 mon.c (mon.2) 16 : audit [INF] from='client.? 172.21.15.78:0/1598175408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-29T17:05:21.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[23704]: audit 2023-07-29T17:05:20.220434+0000 mon.a (mon.0) 534 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-29T17:05:21.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[23704]: audit 2023-07-29T17:05:21.212677+0000 mon.a (mon.0) 535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-29T17:05:21.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:21 smithi078 bash[23704]: cluster 2023-07-29T17:05:21.212808+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2023-07-29T17:05:21.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2023-07-29T17:05:22.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:22 smithi129 bash[23526]: cluster 2023-07-29T17:05:21.145143+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v245: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:22.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:22 smithi078 bash[23704]: cluster 2023-07-29T17:05:21.145143+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v245: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:22.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:22 smithi078 bash[28259]: cluster 2023-07-29T17:05:21.145143+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v245: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:23.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:23 smithi078 bash[28259]: cluster 2023-07-29T17:05:22.233716+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-07-29T17:05:23.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:23 smithi078 bash[28259]: cluster 2023-07-29T17:05:23.145713+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v248: 33 pgs: 13 creating+peering, 4 active+clean, 16 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:23.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:23 smithi078 bash[23704]: cluster 2023-07-29T17:05:22.233716+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-07-29T17:05:23.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:23 smithi078 bash[23704]: cluster 2023-07-29T17:05:23.145713+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v248: 33 pgs: 13 creating+peering, 4 active+clean, 16 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:23.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:23 smithi129 bash[23526]: cluster 2023-07-29T17:05:22.233716+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-07-29T17:05:23.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:23 smithi129 bash[23526]: cluster 2023-07-29T17:05:23.145713+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v248: 33 pgs: 13 creating+peering, 4 active+clean, 16 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:25.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:05:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "20b3bb85-d23b-4865-b8e3-c658ab6667cd", 2023-07-29T17:05:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default", 2023-07-29T17:05:25.003 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "default", 2023-07-29T17:05:25.003 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2023-07-29T17:05:25.003 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2023-07-29T17:05:25.003 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2023-07-29T17:05:25.004 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2023-07-29T17:05:25.004 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "", 2023-07-29T17:05:25.004 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [], 2023-07-29T17:05:25.004 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [], 2023-07-29T17:05:25.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "", 2023-07-29T17:05:25.005 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "6816e162-7db8-4b7f-9381-135762243806", 2023-07-29T17:05:25.005 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2023-07-29T17:05:25.005 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2023-07-29T17:05:25.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:25.006 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:05:25.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2023-07-29T17:05:26.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:26 smithi129 bash[23526]: cluster 2023-07-29T17:05:25.146469+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 33 pgs: 13 creating+peering, 19 active+clean, 1 unknown; 147 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:26.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:26 smithi078 bash[23704]: cluster 2023-07-29T17:05:25.146469+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 33 pgs: 13 creating+peering, 19 active+clean, 1 unknown; 147 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:26.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:26 smithi078 bash[28259]: cluster 2023-07-29T17:05:25.146469+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 33 pgs: 13 creating+peering, 19 active+clean, 1 unknown; 147 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:05:28.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:28 smithi078 bash[28259]: cluster 2023-07-29T17:05:27.147227+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s wr, 3 op/s 2023-07-29T17:05:28.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:28 smithi078 bash[23704]: cluster 2023-07-29T17:05:27.147227+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s wr, 3 op/s 2023-07-29T17:05:28.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:28 smithi129 bash[23526]: cluster 2023-07-29T17:05:27.147227+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s wr, 3 op/s 2023-07-29T17:05:28.759 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:05:28.759 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "c6fdb395-19b9-4efd-8f4e-c129ffaf47c9", 2023-07-29T17:05:28.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2023-07-29T17:05:28.760 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "z.rgw.meta:root", 2023-07-29T17:05:28.760 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "z.rgw.control", 2023-07-29T17:05:28.761 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "z.rgw.log:gc", 2023-07-29T17:05:28.761 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "z.rgw.log:lc", 2023-07-29T17:05:28.761 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "z.rgw.log", 2023-07-29T17:05:28.762 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "z.rgw.log:intent", 2023-07-29T17:05:28.762 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "z.rgw.log:usage", 2023-07-29T17:05:28.762 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "z.rgw.meta:roles", 2023-07-29T17:05:28.762 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "z.rgw.log:reshard", 2023-07-29T17:05:28.762 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2023-07-29T17:05:28.763 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "z.rgw.meta:users.email", 2023-07-29T17:05:28.763 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2023-07-29T17:05:28.763 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2023-07-29T17:05:28.763 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "z.rgw.otp", 2023-07-29T17:05:28.763 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2023-07-29T17:05:28.763 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "", 2023-07-29T17:05:28.764 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "" 2023-07-29T17:05:28.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:05:28.764 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2023-07-29T17:05:28.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-07-29T17:05:28.764 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2023-07-29T17:05:28.764 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "z.rgw.buckets.index", 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "z.rgw.buckets.data" 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2023-07-29T17:05:28.765 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2023-07-29T17:05:28.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:28.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:28.766 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:05:28.766 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "6816e162-7db8-4b7f-9381-135762243806", 2023-07-29T17:05:28.766 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "z.rgw.log:notif" 2023-07-29T17:05:28.766 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:05:29.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2023-07-29T17:05:30.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:30 smithi078 bash[23704]: cluster 2023-07-29T17:05:29.148013+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-29T17:05:30.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:30 smithi078 bash[28259]: cluster 2023-07-29T17:05:29.148013+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-29T17:05:30.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:30 smithi129 bash[23526]: cluster 2023-07-29T17:05:29.148013+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-29T17:05:32.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:32 smithi129 bash[23526]: cluster 2023-07-29T17:05:31.148869+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 33 pgs: 33 active+clean; 1.6 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2023-07-29T17:05:32.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:32 smithi078 bash[23704]: cluster 2023-07-29T17:05:31.148869+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 33 pgs: 33 active+clean; 1.6 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2023-07-29T17:05:32.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:32 smithi078 bash[28259]: cluster 2023-07-29T17:05:31.148869+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 33 pgs: 33 active+clean; 1.6 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2023-07-29T17:05:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:34 smithi129 bash[23526]: cluster 2023-07-29T17:05:33.149598+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 33 pgs: 33 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-29T17:05:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:34 smithi129 bash[23526]: cluster 2023-07-29T17:05:33.233887+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-07-29T17:05:34.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:34 smithi129 bash[23526]: audit 2023-07-29T17:05:33.235565+0000 mon.a (mon.0) 539 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-07-29T17:05:34.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:34 smithi078 bash[28259]: cluster 2023-07-29T17:05:33.149598+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 33 pgs: 33 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-29T17:05:34.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:34 smithi078 bash[28259]: cluster 2023-07-29T17:05:33.233887+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-07-29T17:05:34.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:34 smithi078 bash[28259]: audit 2023-07-29T17:05:33.235565+0000 mon.a (mon.0) 539 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-07-29T17:05:34.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:34 smithi078 bash[23704]: cluster 2023-07-29T17:05:33.149598+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 33 pgs: 33 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-29T17:05:34.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:34 smithi078 bash[23704]: cluster 2023-07-29T17:05:33.233887+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-07-29T17:05:34.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:34 smithi078 bash[23704]: audit 2023-07-29T17:05:33.235565+0000 mon.a (mon.0) 539 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-07-29T17:05:35.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:35 smithi129 bash[23526]: audit 2023-07-29T17:05:34.240941+0000 mon.a (mon.0) 540 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-07-29T17:05:35.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:35 smithi129 bash[23526]: cluster 2023-07-29T17:05:34.241346+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-07-29T17:05:35.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:35 smithi129 bash[23526]: cluster 2023-07-29T17:05:35.150312+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v256: 65 pgs: 6 creating+activating, 5 creating+peering, 21 unknown, 33 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2023-07-29T17:05:35.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:35 smithi078 bash[23704]: audit 2023-07-29T17:05:34.240941+0000 mon.a (mon.0) 540 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-07-29T17:05:35.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:35 smithi078 bash[23704]: cluster 2023-07-29T17:05:34.241346+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-07-29T17:05:35.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:35 smithi078 bash[23704]: cluster 2023-07-29T17:05:35.150312+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v256: 65 pgs: 6 creating+activating, 5 creating+peering, 21 unknown, 33 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2023-07-29T17:05:35.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:35 smithi078 bash[28259]: audit 2023-07-29T17:05:34.240941+0000 mon.a (mon.0) 540 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-07-29T17:05:35.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:35 smithi078 bash[28259]: cluster 2023-07-29T17:05:34.241346+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-07-29T17:05:35.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:35 smithi078 bash[28259]: cluster 2023-07-29T17:05:35.150312+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v256: 65 pgs: 6 creating+activating, 5 creating+peering, 21 unknown, 33 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2023-07-29T17:05:36.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:36 smithi129 bash[23526]: cluster 2023-07-29T17:05:35.251789+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-07-29T17:05:36.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:36 smithi129 bash[23526]: audit 2023-07-29T17:05:35.255987+0000 mon.a (mon.0) 543 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-07-29T17:05:36.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:36 smithi129 bash[23526]: audit 2023-07-29T17:05:35.257152+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:05:36.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:36 smithi129 bash[23526]: audit 2023-07-29T17:05:35.258592+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:05:36.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[23704]: cluster 2023-07-29T17:05:35.251789+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-07-29T17:05:36.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[23704]: audit 2023-07-29T17:05:35.255987+0000 mon.a (mon.0) 543 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-07-29T17:05:36.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[23704]: audit 2023-07-29T17:05:35.257152+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:05:36.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[23704]: audit 2023-07-29T17:05:35.258592+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:05:36.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[28259]: cluster 2023-07-29T17:05:35.251789+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-07-29T17:05:36.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[28259]: audit 2023-07-29T17:05:35.255987+0000 mon.a (mon.0) 543 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-07-29T17:05:36.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[28259]: audit 2023-07-29T17:05:35.257152+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:05:36.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:36 smithi078 bash[28259]: audit 2023-07-29T17:05:35.258592+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:05:37.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:37 smithi129 bash[23526]: audit 2023-07-29T17:05:36.265429+0000 mon.a (mon.0) 546 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-07-29T17:05:37.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:37 smithi129 bash[23526]: cluster 2023-07-29T17:05:36.265655+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-07-29T17:05:37.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:37 smithi129 bash[23526]: cluster 2023-07-29T17:05:37.151080+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v259: 97 pgs: 6 creating+activating, 18 creating+peering, 12 unknown, 61 active+clean; 3.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.2 KiB/s wr, 17 op/s 2023-07-29T17:05:37.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:37 smithi078 bash[23704]: audit 2023-07-29T17:05:36.265429+0000 mon.a (mon.0) 546 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-07-29T17:05:37.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:37 smithi078 bash[23704]: cluster 2023-07-29T17:05:36.265655+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-07-29T17:05:37.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:37 smithi078 bash[23704]: cluster 2023-07-29T17:05:37.151080+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v259: 97 pgs: 6 creating+activating, 18 creating+peering, 12 unknown, 61 active+clean; 3.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.2 KiB/s wr, 17 op/s 2023-07-29T17:05:37.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:37 smithi078 bash[28259]: audit 2023-07-29T17:05:36.265429+0000 mon.a (mon.0) 546 : audit [INF] from='client.? 172.21.15.78:0/2806726305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-07-29T17:05:37.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:37 smithi078 bash[28259]: cluster 2023-07-29T17:05:36.265655+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-07-29T17:05:37.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:37 smithi078 bash[28259]: cluster 2023-07-29T17:05:37.151080+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v259: 97 pgs: 6 creating+activating, 18 creating+peering, 12 unknown, 61 active+clean; 3.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.2 KiB/s wr, 17 op/s 2023-07-29T17:05:38.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:38 smithi129 bash[23526]: cluster 2023-07-29T17:05:37.270902+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-07-29T17:05:38.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:38 smithi078 bash[23704]: cluster 2023-07-29T17:05:37.270902+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-07-29T17:05:38.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:38 smithi078 bash[28259]: cluster 2023-07-29T17:05:37.270902+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-07-29T17:05:39.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:39 smithi129 bash[23526]: cluster 2023-07-29T17:05:38.280259+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-07-29T17:05:39.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:39 smithi129 bash[23526]: audit 2023-07-29T17:05:38.287327+0000 mon.a (mon.0) 550 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-29T17:05:39.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:39 smithi129 bash[23526]: cluster 2023-07-29T17:05:39.151876+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v262: 129 pgs: 6 creating+activating, 18 creating+peering, 43 unknown, 62 active+clean; 3.8 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2023-07-29T17:05:39.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:39 smithi129 bash[23526]: audit 2023-07-29T17:05:39.280099+0000 mon.a (mon.0) 551 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-07-29T17:05:39.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:39 smithi129 bash[23526]: cluster 2023-07-29T17:05:39.280331+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-07-29T17:05:39.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:39 smithi129 bash[23526]: audit 2023-07-29T17:05:39.286415+0000 mon.a (mon.0) 553 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-29T17:05:39.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[23704]: cluster 2023-07-29T17:05:38.280259+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-07-29T17:05:39.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[23704]: audit 2023-07-29T17:05:38.287327+0000 mon.a (mon.0) 550 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-29T17:05:39.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[23704]: cluster 2023-07-29T17:05:39.151876+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v262: 129 pgs: 6 creating+activating, 18 creating+peering, 43 unknown, 62 active+clean; 3.8 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2023-07-29T17:05:39.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[23704]: audit 2023-07-29T17:05:39.280099+0000 mon.a (mon.0) 551 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-07-29T17:05:39.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[23704]: cluster 2023-07-29T17:05:39.280331+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-07-29T17:05:39.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[23704]: audit 2023-07-29T17:05:39.286415+0000 mon.a (mon.0) 553 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-29T17:05:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[28259]: cluster 2023-07-29T17:05:38.280259+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-07-29T17:05:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[28259]: audit 2023-07-29T17:05:38.287327+0000 mon.a (mon.0) 550 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-29T17:05:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[28259]: cluster 2023-07-29T17:05:39.151876+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v262: 129 pgs: 6 creating+activating, 18 creating+peering, 43 unknown, 62 active+clean; 3.8 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2023-07-29T17:05:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[28259]: audit 2023-07-29T17:05:39.280099+0000 mon.a (mon.0) 551 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-07-29T17:05:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[28259]: cluster 2023-07-29T17:05:39.280331+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-07-29T17:05:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:39 smithi078 bash[28259]: audit 2023-07-29T17:05:39.286415+0000 mon.a (mon.0) 553 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-29T17:05:41.492 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:05:41.493 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "dc9aa4db-1c32-44cd-b0ca-cf74d18dc5dd", 2023-07-29T17:05:41.493 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2023-07-29T17:05:41.493 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "fad7ffdd-6aa4-4df6-a289-ec229af69973", 2023-07-29T17:05:41.494 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [], 2023-07-29T17:05:41.494 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2023-07-29T17:05:41.494 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "dc9aa4db-1c32-44cd-b0ca-cf74d18dc5dd", 2023-07-29T17:05:41.494 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2023-07-29T17:05:41.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-07-29T17:05:41.495 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "20b3bb85-d23b-4865-b8e3-c658ab6667cd", 2023-07-29T17:05:41.495 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default", 2023-07-29T17:05:41.495 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "default", 2023-07-29T17:05:41.496 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2023-07-29T17:05:41.496 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2023-07-29T17:05:41.496 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2023-07-29T17:05:41.497 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2023-07-29T17:05:41.497 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "c6fdb395-19b9-4efd-8f4e-c129ffaf47c9", 2023-07-29T17:05:41.497 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2023-07-29T17:05:41.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-07-29T17:05:41.497 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "c6fdb395-19b9-4efd-8f4e-c129ffaf47c9", 2023-07-29T17:05:41.497 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "false", 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2023-07-29T17:05:41.498 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2023-07-29T17:05:41.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2023-07-29T17:05:41.500 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "6816e162-7db8-4b7f-9381-135762243806", 2023-07-29T17:05:41.501 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2023-07-29T17:05:41.501 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2023-07-29T17:05:41.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:41.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:41.501 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:05:41.501 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "c6fdb395-19b9-4efd-8f4e-c129ffaf47c9", 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 1236579586 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "20b3bb85-d23b-4865-b8e3-c658ab6667cd", 2023-07-29T17:05:41.502 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "c6fdb395-19b9-4efd-8f4e-c129ffaf47c9", 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2023-07-29T17:05:41.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2023-07-29T17:05:41.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:05:41.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:05:41.505 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "6816e162-7db8-4b7f-9381-135762243806", 2023-07-29T17:05:41.505 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "r", 2023-07-29T17:05:41.505 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2023-07-29T17:05:41.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:05:41.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:41 smithi129 bash[23526]: audit 2023-07-29T17:05:40.282423+0000 mon.a (mon.0) 554 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-29T17:05:41.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:41 smithi129 bash[23526]: cluster 2023-07-29T17:05:40.282564+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-07-29T17:05:41.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:41 smithi129 bash[23526]: audit 2023-07-29T17:05:40.283847+0000 mon.a (mon.0) 556 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-29T17:05:41.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:41 smithi129 bash[23526]: cluster 2023-07-29T17:05:41.152972+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v265: 129 pgs: 12 creating+peering, 12 unknown, 105 active+clean; 17 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 22 KiB/s wr, 87 op/s 2023-07-29T17:05:41.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[23704]: audit 2023-07-29T17:05:40.282423+0000 mon.a (mon.0) 554 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-29T17:05:41.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[23704]: cluster 2023-07-29T17:05:40.282564+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-07-29T17:05:41.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[23704]: audit 2023-07-29T17:05:40.283847+0000 mon.a (mon.0) 556 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-29T17:05:41.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[23704]: cluster 2023-07-29T17:05:41.152972+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v265: 129 pgs: 12 creating+peering, 12 unknown, 105 active+clean; 17 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 22 KiB/s wr, 87 op/s 2023-07-29T17:05:41.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[28259]: audit 2023-07-29T17:05:40.282423+0000 mon.a (mon.0) 554 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-29T17:05:41.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[28259]: cluster 2023-07-29T17:05:40.282564+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-07-29T17:05:41.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[28259]: audit 2023-07-29T17:05:40.283847+0000 mon.a (mon.0) 556 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-29T17:05:41.670 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:41 smithi078 bash[28259]: cluster 2023-07-29T17:05:41.152972+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v265: 129 pgs: 12 creating+peering, 12 unknown, 105 active+clean; 17 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 22 KiB/s wr, 87 op/s 2023-07-29T17:05:42.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2023-07-29T17:05:42.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:42 smithi078 bash[23704]: audit 2023-07-29T17:05:41.292106+0000 mon.a (mon.0) 557 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-07-29T17:05:42.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:42 smithi078 bash[23704]: cluster 2023-07-29T17:05:41.292190+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-07-29T17:05:42.598 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:42 smithi078 bash[28259]: audit 2023-07-29T17:05:41.292106+0000 mon.a (mon.0) 557 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-07-29T17:05:42.598 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:42 smithi078 bash[28259]: cluster 2023-07-29T17:05:41.292190+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-07-29T17:05:42.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:42 smithi129 bash[23526]: audit 2023-07-29T17:05:41.292106+0000 mon.a (mon.0) 557 : audit [INF] from='client.? 172.21.15.78:0/3863013163' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-07-29T17:05:42.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:42 smithi129 bash[23526]: cluster 2023-07-29T17:05:41.292190+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-07-29T17:05:43.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:43 smithi078 bash[23704]: cluster 2023-07-29T17:05:43.154004+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v267: 129 pgs: 9 creating+peering, 5 unknown, 115 active+clean; 22 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 25 KiB/s wr, 99 op/s 2023-07-29T17:05:43.582 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:43 smithi078 bash[28259]: cluster 2023-07-29T17:05:43.154004+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v267: 129 pgs: 9 creating+peering, 5 unknown, 115 active+clean; 22 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 25 KiB/s wr, 99 op/s 2023-07-29T17:05:43.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:43 smithi129 bash[23526]: cluster 2023-07-29T17:05:43.154004+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v267: 129 pgs: 9 creating+peering, 5 unknown, 115 active+clean; 22 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 25 KiB/s wr, 99 op/s 2023-07-29T17:05:45.618 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled rgw.foo update... 2023-07-29T17:05:46.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: cluster 2023-07-29T17:05:45.155108+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 23 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 22 KiB/s wr, 87 op/s 2023-07-29T17:05:46.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.609677+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.855243+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.866900+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.877629+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.886463+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.887590+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:46.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.895085+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:46.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.904937+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[28259]: audit 2023-07-29T17:05:45.906488+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:46.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: cluster 2023-07-29T17:05:45.155108+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 23 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 22 KiB/s wr, 87 op/s 2023-07-29T17:05:46.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.609677+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.855243+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.866900+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.877629+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.886463+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.887590+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.895085+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:46.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.904937+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:46 smithi078 bash[23704]: audit 2023-07-29T17:05:45.906488+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:46.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: cluster 2023-07-29T17:05:45.155108+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 23 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 22 KiB/s wr, 87 op/s 2023-07-29T17:05:46.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.609677+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.855243+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.866900+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.877629+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.886463+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.887590+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:46.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.895085+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:46.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.904937+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:46.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:46 smithi129 bash[23526]: audit 2023-07-29T17:05:45.906488+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:46.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch apply rgw smpl' 2023-07-29T17:05:47.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:47 smithi129 bash[23526]: audit 2023-07-29T17:05:45.602607+0000 mgr.y (mgr.14144) 287 : audit [DBG] from='client.24590 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:05:47.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:47 smithi129 bash[23526]: cephadm 2023-07-29T17:05:45.605022+0000 mgr.y (mgr.14144) 288 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-29T17:05:47.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:47 smithi129 bash[23526]: cephadm 2023-07-29T17:05:45.878982+0000 mgr.y (mgr.14144) 289 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-29T17:05:47.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:47 smithi129 bash[23526]: cephadm 2023-07-29T17:05:45.908281+0000 mgr.y (mgr.14144) 290 : cephadm [INF] Deploying daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:05:47.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[23704]: audit 2023-07-29T17:05:45.602607+0000 mgr.y (mgr.14144) 287 : audit [DBG] from='client.24590 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:05:47.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[23704]: cephadm 2023-07-29T17:05:45.605022+0000 mgr.y (mgr.14144) 288 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-29T17:05:47.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[23704]: cephadm 2023-07-29T17:05:45.878982+0000 mgr.y (mgr.14144) 289 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-29T17:05:47.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[23704]: cephadm 2023-07-29T17:05:45.908281+0000 mgr.y (mgr.14144) 290 : cephadm [INF] Deploying daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:05:47.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[28259]: audit 2023-07-29T17:05:45.602607+0000 mgr.y (mgr.14144) 287 : audit [DBG] from='client.24590 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:05:47.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[28259]: cephadm 2023-07-29T17:05:45.605022+0000 mgr.y (mgr.14144) 288 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-29T17:05:47.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[28259]: cephadm 2023-07-29T17:05:45.878982+0000 mgr.y (mgr.14144) 289 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-29T17:05:47.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:47 smithi078 bash[28259]: cephadm 2023-07-29T17:05:45.908281+0000 mgr.y (mgr.14144) 290 : cephadm [INF] Deploying daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:05:48.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:48 smithi129 bash[23526]: cluster 2023-07-29T17:05:47.156394+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 KiB/s wr, 57 op/s 2023-07-29T17:05:48.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:48 smithi078 bash[23704]: cluster 2023-07-29T17:05:47.156394+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 KiB/s wr, 57 op/s 2023-07-29T17:05:48.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:48 smithi078 bash[28259]: cluster 2023-07-29T17:05:47.156394+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 KiB/s wr, 57 op/s 2023-07-29T17:05:49.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:49 smithi078 bash[28259]: cluster 2023-07-29T17:05:49.157188+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 12 KiB/s wr, 45 op/s 2023-07-29T17:05:49.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:49 smithi078 bash[23704]: cluster 2023-07-29T17:05:49.157188+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 12 KiB/s wr, 45 op/s 2023-07-29T17:05:49.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:49 smithi129 bash[23526]: cluster 2023-07-29T17:05:49.157188+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 12 KiB/s wr, 45 op/s 2023-07-29T17:05:50.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:50 smithi129 bash[23526]: audit 2023-07-29T17:05:49.468054+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:50.757 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:50 smithi129 bash[23526]: audit 2023-07-29T17:05:49.469314+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:50.757 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:50 smithi129 bash[23526]: audit 2023-07-29T17:05:49.473509+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:50.757 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:50 smithi129 bash[23526]: audit 2023-07-29T17:05:49.478652+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:50.757 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:50 smithi129 bash[23526]: audit 2023-07-29T17:05:49.479787+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:50.757 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:50 smithi129 bash[23526]: cephadm 2023-07-29T17:05:49.480884+0000 mgr.y (mgr.14144) 293 : cephadm [INF] Deploying daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:05:50.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[23704]: audit 2023-07-29T17:05:49.468054+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:50.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[23704]: audit 2023-07-29T17:05:49.469314+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:50.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[23704]: audit 2023-07-29T17:05:49.473509+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:50.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[23704]: audit 2023-07-29T17:05:49.478652+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:50.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[23704]: audit 2023-07-29T17:05:49.479787+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:50.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[23704]: cephadm 2023-07-29T17:05:49.480884+0000 mgr.y (mgr.14144) 293 : cephadm [INF] Deploying daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:05:50.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[28259]: audit 2023-07-29T17:05:49.468054+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:50.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[28259]: audit 2023-07-29T17:05:49.469314+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:50.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[28259]: audit 2023-07-29T17:05:49.473509+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:50.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[28259]: audit 2023-07-29T17:05:49.478652+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:50.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[28259]: audit 2023-07-29T17:05:49.479787+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:50.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:50 smithi078 bash[28259]: cephadm 2023-07-29T17:05:49.480884+0000 mgr.y (mgr.14144) 293 : cephadm [INF] Deploying daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:05:51.621 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled rgw.smpl update... 2023-07-29T17:05:52.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:52 smithi078 bash[23704]: cluster 2023-07-29T17:05:51.158440+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6.6 KiB/s wr, 25 op/s 2023-07-29T17:05:52.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:52 smithi078 bash[23704]: audit 2023-07-29T17:05:51.610338+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:52.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:52 smithi078 bash[23704]: audit 2023-07-29T17:05:52.082191+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:52.412 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:52 smithi129 bash[23526]: cluster 2023-07-29T17:05:51.158440+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6.6 KiB/s wr, 25 op/s 2023-07-29T17:05:52.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:52 smithi129 bash[23526]: audit 2023-07-29T17:05:51.610338+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:52.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:52 smithi129 bash[23526]: audit 2023-07-29T17:05:52.082191+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:52.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:52 smithi078 bash[28259]: cluster 2023-07-29T17:05:51.158440+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6.6 KiB/s wr, 25 op/s 2023-07-29T17:05:52.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:52 smithi078 bash[28259]: audit 2023-07-29T17:05:51.610338+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:52.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:52 smithi078 bash[28259]: audit 2023-07-29T17:05:52.082191+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:52.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph osd pool create foo' 2023-07-29T17:05:53.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:53 smithi078 bash[28259]: audit 2023-07-29T17:05:51.602413+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:05:53.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:53 smithi078 bash[28259]: cephadm 2023-07-29T17:05:51.603762+0000 mgr.y (mgr.14144) 296 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-29T17:05:53.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:53 smithi078 bash[23704]: audit 2023-07-29T17:05:51.602413+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:05:53.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:53 smithi078 bash[23704]: cephadm 2023-07-29T17:05:51.603762+0000 mgr.y (mgr.14144) 296 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-29T17:05:53.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:53 smithi129 bash[23526]: audit 2023-07-29T17:05:51.602413+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:05:53.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:53 smithi129 bash[23526]: cephadm 2023-07-29T17:05:51.603762+0000 mgr.y (mgr.14144) 296 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-29T17:05:54.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:54 smithi078 bash[23704]: cluster 2023-07-29T17:05:53.159431+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.5 KiB/s wr, 36 op/s 2023-07-29T17:05:54.593 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:54 smithi078 bash[28259]: cluster 2023-07-29T17:05:53.159431+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.5 KiB/s wr, 36 op/s 2023-07-29T17:05:54.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:54 smithi129 bash[23526]: cluster 2023-07-29T17:05:53.159431+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.5 KiB/s wr, 36 op/s 2023-07-29T17:05:56.530 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'foo' created 2023-07-29T17:05:56.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:56 smithi129 bash[23526]: cluster 2023-07-29T17:05:55.160510+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 2.7 KiB/s wr, 107 op/s 2023-07-29T17:05:56.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:56 smithi078 bash[23704]: cluster 2023-07-29T17:05:55.160510+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 2.7 KiB/s wr, 107 op/s 2023-07-29T17:05:56.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:56 smithi078 bash[28259]: cluster 2023-07-29T17:05:55.160510+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 2.7 KiB/s wr, 107 op/s 2023-07-29T17:05:57.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'rbd pool init foo' 2023-07-29T17:05:57.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:56.002674+0000 mon.c (mon.2) 17 : audit [INF] from='client.? 172.21.15.78:0/3292725048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:56.003162+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:56.466221+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:56.509123+0000 mon.a (mon.0) 577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: cluster 2023-07-29T17:05:56.509344+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.020751+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.028832+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: cephadm 2023-07-29T17:05:57.032813+0000 mgr.y (mgr.14144) 299 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-29T17:05:57.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.039052+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.040485+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:57.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.045901+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:57.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.051928+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: audit 2023-07-29T17:05:57.053590+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:57.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: cephadm 2023-07-29T17:05:57.054910+0000 mgr.y (mgr.14144) 300 : cephadm [INF] Deploying daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:05:57.680 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[28259]: cluster 2023-07-29T17:05:57.161429+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v275: 161 pgs: 7 creating+peering, 25 unknown, 129 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 204 B/s wr, 369 op/s 2023-07-29T17:05:57.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:56.002674+0000 mon.c (mon.2) 17 : audit [INF] from='client.? 172.21.15.78:0/3292725048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-29T17:05:57.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:56.003162+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-29T17:05:57.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:56.466221+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:56.509123+0000 mon.a (mon.0) 577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: cluster 2023-07-29T17:05:56.509344+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.020751+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.028832+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: cephadm 2023-07-29T17:05:57.032813+0000 mgr.y (mgr.14144) 299 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.039052+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.040485+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:57.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.045901+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:57.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.051928+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: audit 2023-07-29T17:05:57.053590+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:57.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: cephadm 2023-07-29T17:05:57.054910+0000 mgr.y (mgr.14144) 300 : cephadm [INF] Deploying daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:05:57.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:57 smithi078 bash[23704]: cluster 2023-07-29T17:05:57.161429+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v275: 161 pgs: 7 creating+peering, 25 unknown, 129 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 204 B/s wr, 369 op/s 2023-07-29T17:05:57.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:56.002674+0000 mon.c (mon.2) 17 : audit [INF] from='client.? 172.21.15.78:0/3292725048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-29T17:05:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:56.003162+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-29T17:05:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:56.466221+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:56.509123+0000 mon.a (mon.0) 577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-07-29T17:05:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: cluster 2023-07-29T17:05:56.509344+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-07-29T17:05:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.020751+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.028832+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: cephadm 2023-07-29T17:05:57.032813+0000 mgr.y (mgr.14144) 299 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.039052+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.040485+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.045901+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.051928+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: audit 2023-07-29T17:05:57.053590+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:05:57.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: cephadm 2023-07-29T17:05:57.054910+0000 mgr.y (mgr.14144) 300 : cephadm [INF] Deploying daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:05:57.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:57 smithi129 bash[23526]: cluster 2023-07-29T17:05:57.161429+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v275: 161 pgs: 7 creating+peering, 25 unknown, 129 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 204 B/s wr, 369 op/s 2023-07-29T17:05:58.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:58 smithi129 bash[23526]: cluster 2023-07-29T17:05:57.507378+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-07-29T17:05:58.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:58 smithi078 bash[23704]: cluster 2023-07-29T17:05:57.507378+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-07-29T17:05:58.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:58 smithi078 bash[28259]: cluster 2023-07-29T17:05:57.507378+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-07-29T17:05:59.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:59 smithi129 bash[23526]: cluster 2023-07-29T17:05:58.511275+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-07-29T17:05:59.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:05:59 smithi129 bash[23526]: cluster 2023-07-29T17:05:59.162197+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v278: 161 pgs: 7 creating+peering, 23 unknown, 131 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 301 KiB/s rd, 341 B/s wr, 578 op/s 2023-07-29T17:05:59.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:59 smithi078 bash[28259]: cluster 2023-07-29T17:05:58.511275+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-07-29T17:05:59.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:05:59 smithi078 bash[28259]: cluster 2023-07-29T17:05:59.162197+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v278: 161 pgs: 7 creating+peering, 23 unknown, 131 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 301 KiB/s rd, 341 B/s wr, 578 op/s 2023-07-29T17:05:59.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:59 smithi078 bash[23704]: cluster 2023-07-29T17:05:58.511275+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-07-29T17:05:59.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:05:59 smithi078 bash[23704]: cluster 2023-07-29T17:05:59.162197+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v278: 161 pgs: 7 creating+peering, 23 unknown, 131 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 301 KiB/s rd, 341 B/s wr, 578 op/s 2023-07-29T17:06:02.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: audit 2023-07-29T17:06:00.758818+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:02.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: audit 2023-07-29T17:06:00.760008+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:06:02.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: audit 2023-07-29T17:06:00.765438+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:06:02.130 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: audit 2023-07-29T17:06:00.773750+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:02.130 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: audit 2023-07-29T17:06:00.774879+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:06:02.130 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: cephadm 2023-07-29T17:06:00.775954+0000 mgr.y (mgr.14144) 303 : cephadm [INF] Deploying daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:06:02.130 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[28259]: cluster 2023-07-29T17:06:01.163674+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v279: 161 pgs: 7 creating+peering, 154 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 261 KiB/s rd, 341 B/s wr, 480 op/s 2023-07-29T17:06:02.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: audit 2023-07-29T17:06:00.758818+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:02.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: audit 2023-07-29T17:06:00.760008+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:06:02.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: audit 2023-07-29T17:06:00.765438+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:06:02.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: audit 2023-07-29T17:06:00.773750+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:02.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: audit 2023-07-29T17:06:00.774879+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:06:02.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: cephadm 2023-07-29T17:06:00.775954+0000 mgr.y (mgr.14144) 303 : cephadm [INF] Deploying daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:06:02.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:01 smithi078 bash[23704]: cluster 2023-07-29T17:06:01.163674+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v279: 161 pgs: 7 creating+peering, 154 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 261 KiB/s rd, 341 B/s wr, 480 op/s 2023-07-29T17:06:02.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: audit 2023-07-29T17:06:00.758818+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:02.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: audit 2023-07-29T17:06:00.760008+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:06:02.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: audit 2023-07-29T17:06:00.765438+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-29T17:06:02.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: audit 2023-07-29T17:06:00.773750+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:02.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: audit 2023-07-29T17:06:00.774879+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:06:02.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: cephadm 2023-07-29T17:06:00.775954+0000 mgr.y (mgr.14144) 303 : cephadm [INF] Deploying daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:06:02.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:01 smithi129 bash[23526]: cluster 2023-07-29T17:06:01.163674+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v279: 161 pgs: 7 creating+peering, 154 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 261 KiB/s rd, 341 B/s wr, 480 op/s 2023-07-29T17:06:02.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:02 smithi129 bash[23526]: audit 2023-07-29T17:06:02.042585+0000 mon.c (mon.2) 18 : audit [INF] from='client.? 172.21.15.78:0/608944026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-29T17:06:02.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:02 smithi129 bash[23526]: audit 2023-07-29T17:06:02.043208+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-29T17:06:03.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:02 smithi078 bash[23704]: audit 2023-07-29T17:06:02.042585+0000 mon.c (mon.2) 18 : audit [INF] from='client.? 172.21.15.78:0/608944026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-29T17:06:03.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:02 smithi078 bash[23704]: audit 2023-07-29T17:06:02.043208+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-29T17:06:03.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:02 smithi078 bash[28259]: audit 2023-07-29T17:06:02.042585+0000 mon.c (mon.2) 18 : audit [INF] from='client.? 172.21.15.78:0/608944026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-29T17:06:03.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:02 smithi078 bash[28259]: audit 2023-07-29T17:06:02.043208+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-29T17:06:04.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:03 smithi129 bash[23526]: audit 2023-07-29T17:06:02.790905+0000 mon.a (mon.0) 594 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-07-29T17:06:04.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:03 smithi129 bash[23526]: cluster 2023-07-29T17:06:02.791045+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-07-29T17:06:04.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:03 smithi129 bash[23526]: cluster 2023-07-29T17:06:03.164316+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 58 op/s 2023-07-29T17:06:04.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:03 smithi129 bash[23526]: audit 2023-07-29T17:06:03.745138+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:04.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[28259]: audit 2023-07-29T17:06:02.790905+0000 mon.a (mon.0) 594 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-07-29T17:06:04.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[28259]: cluster 2023-07-29T17:06:02.791045+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-07-29T17:06:04.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[28259]: cluster 2023-07-29T17:06:03.164316+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 58 op/s 2023-07-29T17:06:04.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[28259]: audit 2023-07-29T17:06:03.745138+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:04.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[23704]: audit 2023-07-29T17:06:02.790905+0000 mon.a (mon.0) 594 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-07-29T17:06:04.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[23704]: cluster 2023-07-29T17:06:02.791045+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-07-29T17:06:04.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[23704]: cluster 2023-07-29T17:06:03.164316+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 58 op/s 2023-07-29T17:06:04.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:03 smithi078 bash[23704]: audit 2023-07-29T17:06:03.745138+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:05.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:04 smithi129 bash[23526]: cluster 2023-07-29T17:06:03.803448+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-07-29T17:06:05.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:04 smithi078 bash[23704]: cluster 2023-07-29T17:06:03.803448+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-07-29T17:06:05.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:04 smithi078 bash[28259]: cluster 2023-07-29T17:06:03.803448+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-07-29T17:06:05.736 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch apply iscsi foo u p' 2023-07-29T17:06:06.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:05 smithi129 bash[23526]: cluster 2023-07-29T17:06:04.809451+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-07-29T17:06:06.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:05 smithi129 bash[23526]: cluster 2023-07-29T17:06:05.165149+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2023-07-29T17:06:06.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:05 smithi078 bash[28259]: cluster 2023-07-29T17:06:04.809451+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-07-29T17:06:06.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:05 smithi078 bash[28259]: cluster 2023-07-29T17:06:05.165149+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2023-07-29T17:06:06.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:05 smithi078 bash[23704]: cluster 2023-07-29T17:06:04.809451+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-07-29T17:06:06.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:05 smithi078 bash[23704]: cluster 2023-07-29T17:06:05.165149+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2023-07-29T17:06:07.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:07 smithi078 bash[23704]: cluster 2023-07-29T17:06:07.166652+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 271 KiB/s rd, 682 B/s wr, 480 op/s 2023-07-29T17:06:07.644 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:07 smithi078 bash[28259]: cluster 2023-07-29T17:06:07.166652+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 271 KiB/s rd, 682 B/s wr, 480 op/s 2023-07-29T17:06:07.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:07 smithi129 bash[23526]: cluster 2023-07-29T17:06:07.166652+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 271 KiB/s rd, 682 B/s wr, 480 op/s 2023-07-29T17:06:09.267 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled iscsi.foo update... 2023-07-29T17:06:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:08.647344+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:08.742418+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:08.748514+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:08.759458+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: cluster 2023-07-29T17:06:09.167614+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 641 B/s wr, 452 op/s 2023-07-29T17:06:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:09.257592+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:09.266077+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:09.277648+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:06:09.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:09.283893+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-29T17:06:09.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:09 smithi129 bash[23526]: audit 2023-07-29T17:06:09.292361+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:06:09.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:08.647344+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:08.742418+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:08.748514+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:08.759458+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: cluster 2023-07-29T17:06:09.167614+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 641 B/s wr, 452 op/s 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:09.257592+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:09.266077+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:09.277648+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:06:09.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:09.283893+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-29T17:06:09.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[23704]: audit 2023-07-29T17:06:09.292361+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:06:09.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:08.647344+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:08.742418+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:08.748514+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:08.759458+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: cluster 2023-07-29T17:06:09.167614+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 641 B/s wr, 452 op/s 2023-07-29T17:06:09.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:09.257592+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:09.266077+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:09.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:09.277648+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:06:09.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:09.283893+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-29T17:06:09.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:09 smithi078 bash[28259]: audit 2023-07-29T17:06:09.292361+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:06:09.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'sleep 120' 2023-07-29T17:06:11.058 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[23704]: audit 2023-07-29T17:06:09.251108+0000 mgr.y (mgr.14144) 309 : audit [DBG] from='client.14781 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:06:11.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[23704]: cephadm 2023-07-29T17:06:09.251873+0000 mgr.y (mgr.14144) 310 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-07-29T17:06:11.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[23704]: cephadm 2023-07-29T17:06:09.269299+0000 mgr.y (mgr.14144) 311 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-07-29T17:06:11.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[23704]: cephadm 2023-07-29T17:06:09.293478+0000 mgr.y (mgr.14144) 312 : cephadm [INF] Deploying daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:06:11.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[23704]: cluster 2023-07-29T17:06:10.295929+0000 mon.a (mon.0) 608 : cluster [DBG] mgrmap e15: y(active, since 7m), standbys: x 2023-07-29T17:06:11.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[28259]: audit 2023-07-29T17:06:09.251108+0000 mgr.y (mgr.14144) 309 : audit [DBG] from='client.14781 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:06:11.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[28259]: cephadm 2023-07-29T17:06:09.251873+0000 mgr.y (mgr.14144) 310 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-07-29T17:06:11.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[28259]: cephadm 2023-07-29T17:06:09.269299+0000 mgr.y (mgr.14144) 311 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-07-29T17:06:11.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[28259]: cephadm 2023-07-29T17:06:09.293478+0000 mgr.y (mgr.14144) 312 : cephadm [INF] Deploying daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:06:11.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:10 smithi078 bash[28259]: cluster 2023-07-29T17:06:10.295929+0000 mon.a (mon.0) 608 : cluster [DBG] mgrmap e15: y(active, since 7m), standbys: x 2023-07-29T17:06:11.080 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:10 smithi129 bash[23526]: audit 2023-07-29T17:06:09.251108+0000 mgr.y (mgr.14144) 309 : audit [DBG] from='client.14781 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:06:11.081 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:10 smithi129 bash[23526]: cephadm 2023-07-29T17:06:09.251873+0000 mgr.y (mgr.14144) 310 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-07-29T17:06:11.081 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:10 smithi129 bash[23526]: cephadm 2023-07-29T17:06:09.269299+0000 mgr.y (mgr.14144) 311 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-07-29T17:06:11.081 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:10 smithi129 bash[23526]: cephadm 2023-07-29T17:06:09.293478+0000 mgr.y (mgr.14144) 312 : cephadm [INF] Deploying daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:06:11.081 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:10 smithi129 bash[23526]: cluster 2023-07-29T17:06:10.295929+0000 mon.a (mon.0) 608 : cluster [DBG] mgrmap e15: y(active, since 7m), standbys: x 2023-07-29T17:06:11.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:11 smithi129 bash[23526]: cluster 2023-07-29T17:06:11.168943+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 511 B/s wr, 501 op/s 2023-07-29T17:06:12.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:11 smithi078 bash[23704]: cluster 2023-07-29T17:06:11.168943+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 511 B/s wr, 501 op/s 2023-07-29T17:06:12.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:11 smithi078 bash[28259]: cluster 2023-07-29T17:06:11.168943+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 511 B/s wr, 501 op/s 2023-07-29T17:06:13.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:13 smithi129 bash[23526]: audit 2023-07-29T17:06:12.138773+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:13.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:13 smithi078 bash[23704]: audit 2023-07-29T17:06:12.138773+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:13.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:13 smithi078 bash[28259]: audit 2023-07-29T17:06:12.138773+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:14.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:14 smithi129 bash[23526]: cluster 2023-07-29T17:06:13.170015+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 248 KiB/s rd, 437 B/s wr, 433 op/s 2023-07-29T17:06:14.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:14 smithi078 bash[23704]: cluster 2023-07-29T17:06:13.170015+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 248 KiB/s rd, 437 B/s wr, 433 op/s 2023-07-29T17:06:14.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:14 smithi078 bash[28259]: cluster 2023-07-29T17:06:13.170015+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 248 KiB/s rd, 437 B/s wr, 433 op/s 2023-07-29T17:06:15.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:15 smithi129 bash[23526]: audit 2023-07-29T17:06:14.489962+0000 mon.b (mon.1) 25 : audit [DBG] from='client.? 172.21.15.129:0/1851465167' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:06:15.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:15 smithi078 bash[28259]: audit 2023-07-29T17:06:14.489962+0000 mon.b (mon.1) 25 : audit [DBG] from='client.? 172.21.15.129:0/1851465167' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:06:15.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:15 smithi078 bash[23704]: audit 2023-07-29T17:06:14.489962+0000 mon.b (mon.1) 25 : audit [DBG] from='client.? 172.21.15.129:0/1851465167' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:06:16.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:16 smithi078 bash[28259]: cluster 2023-07-29T17:06:15.171166+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 188 KiB/s rd, 395 B/s wr, 328 op/s 2023-07-29T17:06:16.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:16 smithi078 bash[23704]: cluster 2023-07-29T17:06:15.171166+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 188 KiB/s rd, 395 B/s wr, 328 op/s 2023-07-29T17:06:16.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:16 smithi129 bash[23526]: cluster 2023-07-29T17:06:15.171166+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 188 KiB/s rd, 395 B/s wr, 328 op/s 2023-07-29T17:06:17.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: cluster 2023-07-29T17:06:16.460656+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-07-29T17:06:17.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: cluster 2023-07-29T17:06:17.172149+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 102 B/s wr, 118 op/s 2023-07-29T17:06:17.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.371926+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.379937+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.390194+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:06:17.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.394238+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:06:17.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.403719+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.408034+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:06:17.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.423308+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:17 smithi129 bash[23526]: audit 2023-07-29T17:06:17.431651+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: cluster 2023-07-29T17:06:16.460656+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-07-29T17:06:17.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: cluster 2023-07-29T17:06:17.172149+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 102 B/s wr, 118 op/s 2023-07-29T17:06:17.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.371926+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.379937+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.390194+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:06:17.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.394238+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:06:17.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.403719+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.408034+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:06:17.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.423308+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[28259]: audit 2023-07-29T17:06:17.431651+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: cluster 2023-07-29T17:06:16.460656+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-07-29T17:06:17.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: cluster 2023-07-29T17:06:17.172149+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 102 B/s wr, 118 op/s 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.371926+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.379937+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.390194+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.394238+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.403719+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.408034+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.423308+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:17.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:17 smithi078 bash[23704]: audit 2023-07-29T17:06:17.431651+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:06:18.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:18 smithi129 bash[23526]: audit 2023-07-29T17:06:17.391396+0000 mgr.y (mgr.14144) 317 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:06:18.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:18 smithi129 bash[23526]: cephadm 2023-07-29T17:06:17.393657+0000 mgr.y (mgr.14144) 318 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.129:5000 to Dashboard 2023-07-29T17:06:18.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:18 smithi129 bash[23526]: audit 2023-07-29T17:06:17.395015+0000 mgr.y (mgr.14144) 319 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:06:18.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:18 smithi129 bash[23526]: audit 2023-07-29T17:06:17.408877+0000 mgr.y (mgr.14144) 320 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:06:18.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[28259]: audit 2023-07-29T17:06:17.391396+0000 mgr.y (mgr.14144) 317 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:06:18.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[28259]: cephadm 2023-07-29T17:06:17.393657+0000 mgr.y (mgr.14144) 318 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.129:5000 to Dashboard 2023-07-29T17:06:18.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[28259]: audit 2023-07-29T17:06:17.395015+0000 mgr.y (mgr.14144) 319 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:06:18.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[28259]: audit 2023-07-29T17:06:17.408877+0000 mgr.y (mgr.14144) 320 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:06:18.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[23704]: audit 2023-07-29T17:06:17.391396+0000 mgr.y (mgr.14144) 317 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:06:18.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[23704]: cephadm 2023-07-29T17:06:17.393657+0000 mgr.y (mgr.14144) 318 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.129:5000 to Dashboard 2023-07-29T17:06:18.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[23704]: audit 2023-07-29T17:06:17.395015+0000 mgr.y (mgr.14144) 319 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:06:18.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:18 smithi078 bash[23704]: audit 2023-07-29T17:06:17.408877+0000 mgr.y (mgr.14144) 320 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:06:19.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:19 smithi129 bash[23526]: cluster 2023-07-29T17:06:19.173026+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 102 B/s wr, 118 op/s 2023-07-29T17:06:19.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:19 smithi078 bash[23704]: cluster 2023-07-29T17:06:19.173026+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 102 B/s wr, 118 op/s 2023-07-29T17:06:19.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:19 smithi078 bash[28259]: cluster 2023-07-29T17:06:19.173026+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 102 B/s wr, 118 op/s 2023-07-29T17:06:22.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:22 smithi129 bash[23526]: cluster 2023-07-29T17:06:21.174425+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2023-07-29T17:06:22.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:22 smithi078 bash[23704]: cluster 2023-07-29T17:06:21.174425+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2023-07-29T17:06:22.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:22 smithi078 bash[28259]: cluster 2023-07-29T17:06:21.174425+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2023-07-29T17:06:23.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:23 smithi129 bash[23526]: cluster 2023-07-29T17:06:23.175561+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 102 B/s wr, 1 op/s 2023-07-29T17:06:23.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:23 smithi078 bash[23704]: cluster 2023-07-29T17:06:23.175561+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 102 B/s wr, 1 op/s 2023-07-29T17:06:23.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:23 smithi078 bash[28259]: cluster 2023-07-29T17:06:23.175561+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 102 B/s wr, 1 op/s 2023-07-29T17:06:26.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:26 smithi129 bash[23526]: cluster 2023-07-29T17:06:25.176831+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 102 B/s wr, 1 op/s 2023-07-29T17:06:26.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:26 smithi078 bash[23704]: cluster 2023-07-29T17:06:25.176831+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 102 B/s wr, 1 op/s 2023-07-29T17:06:26.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:26 smithi078 bash[28259]: cluster 2023-07-29T17:06:25.176831+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 102 B/s wr, 1 op/s 2023-07-29T17:06:27.913 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:27 smithi129 bash[23526]: cluster 2023-07-29T17:06:27.178331+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 B/s wr, 2 op/s 2023-07-29T17:06:27.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:27 smithi078 bash[23704]: cluster 2023-07-29T17:06:27.178331+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 B/s wr, 2 op/s 2023-07-29T17:06:27.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:27 smithi078 bash[28259]: cluster 2023-07-29T17:06:27.178331+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 B/s wr, 2 op/s 2023-07-29T17:06:29.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:29 smithi129 bash[23526]: cluster 2023-07-29T17:06:29.179291+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:29.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:29 smithi078 bash[23704]: cluster 2023-07-29T17:06:29.179291+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:29.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:29 smithi078 bash[28259]: cluster 2023-07-29T17:06:29.179291+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:31.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:31 smithi129 bash[23526]: cluster 2023-07-29T17:06:31.180752+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:31.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:31 smithi078 bash[23704]: cluster 2023-07-29T17:06:31.180752+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:31 smithi078 bash[28259]: cluster 2023-07-29T17:06:31.180752+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:33.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:33 smithi129 bash[23526]: cluster 2023-07-29T17:06:33.181773+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:33.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:33 smithi078 bash[23704]: cluster 2023-07-29T17:06:33.181773+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:33.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:33 smithi078 bash[28259]: cluster 2023-07-29T17:06:33.181773+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:35.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:35 smithi129 bash[23526]: audit 2023-07-29T17:06:35.070829+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2023-07-29T17:06:35.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:35 smithi129 bash[23526]: audit 2023-07-29T17:06:35.071615+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2023-07-29T17:06:35.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:35 smithi129 bash[23526]: audit 2023-07-29T17:06:35.072340+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2023-07-29T17:06:35.415 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:35 smithi129 bash[23526]: audit 2023-07-29T17:06:35.073030+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-07-29T17:06:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[23704]: audit 2023-07-29T17:06:35.070829+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2023-07-29T17:06:35.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[23704]: audit 2023-07-29T17:06:35.071615+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2023-07-29T17:06:35.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[23704]: audit 2023-07-29T17:06:35.072340+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2023-07-29T17:06:35.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[23704]: audit 2023-07-29T17:06:35.073030+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-07-29T17:06:35.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[28259]: audit 2023-07-29T17:06:35.070829+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2023-07-29T17:06:35.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[28259]: audit 2023-07-29T17:06:35.071615+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2023-07-29T17:06:35.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[28259]: audit 2023-07-29T17:06:35.072340+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2023-07-29T17:06:35.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:35 smithi078 bash[28259]: audit 2023-07-29T17:06:35.073030+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-07-29T17:06:36.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: audit 2023-07-29T17:06:35.131921+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2023-07-29T17:06:36.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: audit 2023-07-29T17:06:35.132170+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2023-07-29T17:06:36.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: audit 2023-07-29T17:06:35.132445+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2023-07-29T17:06:36.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: audit 2023-07-29T17:06:35.132596+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-07-29T17:06:36.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: cluster 2023-07-29T17:06:35.132686+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-07-29T17:06:36.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: cluster 2023-07-29T17:06:35.182624+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-29T17:06:36.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: audit 2023-07-29T17:06:35.263871+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:06:36.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[23704]: audit 2023-07-29T17:06:35.264468+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:06:36.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: audit 2023-07-29T17:06:35.131921+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: audit 2023-07-29T17:06:35.132170+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: audit 2023-07-29T17:06:35.132445+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: audit 2023-07-29T17:06:35.132596+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: cluster 2023-07-29T17:06:35.132686+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: cluster 2023-07-29T17:06:35.182624+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: audit 2023-07-29T17:06:35.263871+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:06:36.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:36 smithi078 bash[28259]: audit 2023-07-29T17:06:35.264468+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:06:36.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: audit 2023-07-29T17:06:35.131921+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2023-07-29T17:06:36.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: audit 2023-07-29T17:06:35.132170+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2023-07-29T17:06:36.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: audit 2023-07-29T17:06:35.132445+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2023-07-29T17:06:36.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: audit 2023-07-29T17:06:35.132596+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-07-29T17:06:36.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: cluster 2023-07-29T17:06:35.132686+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-07-29T17:06:36.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: cluster 2023-07-29T17:06:35.182624+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-29T17:06:36.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: audit 2023-07-29T17:06:35.263871+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:06:36.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:36 smithi129 bash[23526]: audit 2023-07-29T17:06:35.264468+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:06:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:37 smithi078 bash[28259]: cluster 2023-07-29T17:06:36.151429+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-07-29T17:06:37.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:37 smithi078 bash[23704]: cluster 2023-07-29T17:06:36.151429+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-07-29T17:06:37.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:37 smithi129 bash[23526]: cluster 2023-07-29T17:06:36.151429+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-07-29T17:06:38.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:38 smithi078 bash[28259]: cluster 2023-07-29T17:06:37.183745+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v303: 161 pgs: 3 peering, 158 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:38.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:38 smithi078 bash[23704]: cluster 2023-07-29T17:06:37.183745+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v303: 161 pgs: 3 peering, 158 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:38.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:38 smithi129 bash[23526]: cluster 2023-07-29T17:06:37.183745+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v303: 161 pgs: 3 peering, 158 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:39.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:39 smithi129 bash[23526]: cluster 2023-07-29T17:06:39.184583+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v304: 161 pgs: 3 peering, 158 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:39.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:39 smithi078 bash[23704]: cluster 2023-07-29T17:06:39.184583+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v304: 161 pgs: 3 peering, 158 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:39.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:39 smithi078 bash[28259]: cluster 2023-07-29T17:06:39.184583+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v304: 161 pgs: 3 peering, 158 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:41.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:41 smithi129 bash[23526]: cluster 2023-07-29T17:06:41.186033+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:41.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:41 smithi078 bash[28259]: cluster 2023-07-29T17:06:41.186033+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:41.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:41 smithi078 bash[23704]: cluster 2023-07-29T17:06:41.186033+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:43.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:43 smithi129 bash[23526]: cluster 2023-07-29T17:06:43.187100+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:43.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:43 smithi078 bash[28259]: cluster 2023-07-29T17:06:43.187100+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:43.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:43 smithi078 bash[23704]: cluster 2023-07-29T17:06:43.187100+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:45.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:45 smithi129 bash[23526]: cluster 2023-07-29T17:06:45.188280+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:45.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:45 smithi078 bash[23704]: cluster 2023-07-29T17:06:45.188280+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:45.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:45 smithi078 bash[28259]: cluster 2023-07-29T17:06:45.188280+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:47.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:47 smithi129 bash[23526]: cluster 2023-07-29T17:06:47.189719+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:47.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:47 smithi078 bash[23704]: cluster 2023-07-29T17:06:47.189719+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:47.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:47 smithi078 bash[28259]: cluster 2023-07-29T17:06:47.189719+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:49.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:49 smithi129 bash[23526]: cluster 2023-07-29T17:06:49.190625+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:49.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:49 smithi078 bash[23704]: cluster 2023-07-29T17:06:49.190625+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:49.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:49 smithi078 bash[28259]: cluster 2023-07-29T17:06:49.190625+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:51.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:51 smithi129 bash[23526]: cluster 2023-07-29T17:06:51.192067+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:51.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:51 smithi078 bash[23704]: cluster 2023-07-29T17:06:51.192067+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:51.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:51 smithi078 bash[28259]: cluster 2023-07-29T17:06:51.192067+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2023-07-29T17:06:53.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:53 smithi129 bash[23526]: cluster 2023-07-29T17:06:53.193110+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:53.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:53 smithi078 bash[23704]: cluster 2023-07-29T17:06:53.193110+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:53.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:53 smithi078 bash[28259]: cluster 2023-07-29T17:06:53.193110+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:55.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:55 smithi129 bash[23526]: cluster 2023-07-29T17:06:55.194295+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:55.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:55 smithi078 bash[23704]: cluster 2023-07-29T17:06:55.194295+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:55.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:55 smithi078 bash[28259]: cluster 2023-07-29T17:06:55.194295+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:57.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:57 smithi129 bash[23526]: cluster 2023-07-29T17:06:57.195732+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:57.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:57 smithi078 bash[23704]: cluster 2023-07-29T17:06:57.195732+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:57.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:57 smithi078 bash[28259]: cluster 2023-07-29T17:06:57.195732+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:06:59.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:06:59 smithi129 bash[23526]: cluster 2023-07-29T17:06:59.196571+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:59.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:06:59 smithi078 bash[23704]: cluster 2023-07-29T17:06:59.196571+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:06:59.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:06:59 smithi078 bash[28259]: cluster 2023-07-29T17:06:59.196571+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:01.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:01 smithi129 bash[23526]: cluster 2023-07-29T17:07:01.197958+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:01.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:01 smithi078 bash[23704]: cluster 2023-07-29T17:07:01.197958+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:01.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:01 smithi078 bash[28259]: cluster 2023-07-29T17:07:01.197958+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:03.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:03 smithi129 bash[23526]: cluster 2023-07-29T17:07:03.199139+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:03.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:03 smithi078 bash[23704]: cluster 2023-07-29T17:07:03.199139+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:03.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:03 smithi078 bash[28259]: cluster 2023-07-29T17:07:03.199139+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:05.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:05 smithi129 bash[23526]: cluster 2023-07-29T17:07:05.200365+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:05.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:05 smithi078 bash[23704]: cluster 2023-07-29T17:07:05.200365+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:05.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:05 smithi078 bash[28259]: cluster 2023-07-29T17:07:05.200365+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:07.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:07 smithi129 bash[23526]: cluster 2023-07-29T17:07:07.201791+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:07.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:07 smithi078 bash[23704]: cluster 2023-07-29T17:07:07.201791+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:07.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:07 smithi078 bash[28259]: cluster 2023-07-29T17:07:07.201791+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:09.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:09 smithi129 bash[23526]: cluster 2023-07-29T17:07:09.202727+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:09.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:09 smithi078 bash[23704]: cluster 2023-07-29T17:07:09.202727+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:09.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:09 smithi078 bash[28259]: cluster 2023-07-29T17:07:09.202727+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:11.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:11 smithi129 bash[23526]: cluster 2023-07-29T17:07:11.204176+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:11.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:11 smithi078 bash[23704]: cluster 2023-07-29T17:07:11.204176+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:11.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:11 smithi078 bash[28259]: cluster 2023-07-29T17:07:11.204176+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:13.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:13 smithi129 bash[23526]: cluster 2023-07-29T17:07:13.205300+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:13.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:13 smithi078 bash[23704]: cluster 2023-07-29T17:07:13.205300+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:13.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:13 smithi078 bash[28259]: cluster 2023-07-29T17:07:13.205300+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:15 smithi129 bash[23526]: cluster 2023-07-29T17:07:15.206538+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:15.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:15 smithi078 bash[23704]: cluster 2023-07-29T17:07:15.206538+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:15.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:15 smithi078 bash[28259]: cluster 2023-07-29T17:07:15.206538+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:17.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:17 smithi129 bash[23526]: cluster 2023-07-29T17:07:17.207870+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:17.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:17 smithi078 bash[23704]: cluster 2023-07-29T17:07:17.207870+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:17.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:17 smithi078 bash[28259]: cluster 2023-07-29T17:07:17.207870+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:19.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:18 smithi129 bash[23526]: audit 2023-07-29T17:07:17.712369+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:07:19.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:18 smithi078 bash[23704]: audit 2023-07-29T17:07:17.712369+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:07:19.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:18 smithi078 bash[28259]: audit 2023-07-29T17:07:17.712369+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:07:20.165 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:19 smithi129 bash[23526]: cluster 2023-07-29T17:07:19.208708+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:20.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:19 smithi078 bash[23704]: cluster 2023-07-29T17:07:19.208708+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:20.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:19 smithi078 bash[28259]: cluster 2023-07-29T17:07:19.208708+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:21.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:21 smithi129 bash[23526]: cluster 2023-07-29T17:07:21.210133+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:21.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:21 smithi078 bash[28259]: cluster 2023-07-29T17:07:21.210133+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:21.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:21 smithi078 bash[23704]: cluster 2023-07-29T17:07:21.210133+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:23.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:23 smithi129 bash[23526]: cluster 2023-07-29T17:07:23.211237+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:23.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:23 smithi078 bash[23704]: cluster 2023-07-29T17:07:23.211237+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:23.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:23 smithi078 bash[28259]: cluster 2023-07-29T17:07:23.211237+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:25.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:25 smithi129 bash[23526]: cluster 2023-07-29T17:07:25.212426+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:25.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:25 smithi078 bash[28259]: cluster 2023-07-29T17:07:25.212426+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:25.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:25 smithi078 bash[23704]: cluster 2023-07-29T17:07:25.212426+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:27.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:27 smithi129 bash[23526]: cluster 2023-07-29T17:07:27.213728+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:27.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:27 smithi078 bash[28259]: cluster 2023-07-29T17:07:27.213728+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:27.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:27 smithi078 bash[23704]: cluster 2023-07-29T17:07:27.213728+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:29.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:29 smithi129 bash[23526]: cluster 2023-07-29T17:07:29.214694+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:29.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:29 smithi078 bash[28259]: cluster 2023-07-29T17:07:29.214694+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:29.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:29 smithi078 bash[23704]: cluster 2023-07-29T17:07:29.214694+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:31.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:31 smithi129 bash[23526]: cluster 2023-07-29T17:07:31.216211+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v330: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:31.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:31 smithi078 bash[23704]: cluster 2023-07-29T17:07:31.216211+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v330: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:31.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:31 smithi078 bash[28259]: cluster 2023-07-29T17:07:31.216211+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v330: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:33.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:33 smithi129 bash[23526]: cluster 2023-07-29T17:07:33.217315+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v331: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:33.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:33 smithi078 bash[28259]: cluster 2023-07-29T17:07:33.217315+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v331: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:33.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:33 smithi078 bash[23704]: cluster 2023-07-29T17:07:33.217315+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v331: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:35.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:35 smithi129 bash[23526]: cluster 2023-07-29T17:07:35.218441+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:35.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:35 smithi129 bash[23526]: audit 2023-07-29T17:07:35.279154+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:07:35.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:35 smithi129 bash[23526]: audit 2023-07-29T17:07:35.280007+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:07:35.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:35 smithi078 bash[23704]: cluster 2023-07-29T17:07:35.218441+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:35.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:35 smithi078 bash[23704]: audit 2023-07-29T17:07:35.279154+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:07:35.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:35 smithi078 bash[23704]: audit 2023-07-29T17:07:35.280007+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:07:35.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:35 smithi078 bash[28259]: cluster 2023-07-29T17:07:35.218441+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:35.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:35 smithi078 bash[28259]: audit 2023-07-29T17:07:35.279154+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:07:35.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:35 smithi078 bash[28259]: audit 2023-07-29T17:07:35.280007+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:07:37.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:37 smithi129 bash[23526]: cluster 2023-07-29T17:07:37.219770+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v333: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:37.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:37 smithi078 bash[23704]: cluster 2023-07-29T17:07:37.219770+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v333: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:37.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:37 smithi078 bash[28259]: cluster 2023-07-29T17:07:37.219770+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v333: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:39.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:39 smithi129 bash[23526]: cluster 2023-07-29T17:07:39.220656+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v334: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:39.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:39 smithi078 bash[28259]: cluster 2023-07-29T17:07:39.220656+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v334: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:39.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:39 smithi078 bash[23704]: cluster 2023-07-29T17:07:39.220656+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v334: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:41.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:41 smithi129 bash[23526]: cluster 2023-07-29T17:07:41.222138+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v335: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:41.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:41 smithi078 bash[23704]: cluster 2023-07-29T17:07:41.222138+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v335: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:41.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:41 smithi078 bash[28259]: cluster 2023-07-29T17:07:41.222138+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v335: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:43.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:43 smithi129 bash[23526]: cluster 2023-07-29T17:07:43.223172+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v336: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:43.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:43 smithi078 bash[23704]: cluster 2023-07-29T17:07:43.223172+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v336: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:43.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:43 smithi078 bash[28259]: cluster 2023-07-29T17:07:43.223172+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v336: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:45.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:45 smithi129 bash[23526]: cluster 2023-07-29T17:07:45.224370+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v337: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:45.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:45 smithi078 bash[28259]: cluster 2023-07-29T17:07:45.224370+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v337: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:45.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:45 smithi078 bash[23704]: cluster 2023-07-29T17:07:45.224370+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v337: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:47.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:47 smithi129 bash[23526]: cluster 2023-07-29T17:07:47.225676+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v338: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:47.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:47 smithi078 bash[28259]: cluster 2023-07-29T17:07:47.225676+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v338: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:47.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:47 smithi078 bash[23704]: cluster 2023-07-29T17:07:47.225676+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v338: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:49.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:49 smithi129 bash[23526]: cluster 2023-07-29T17:07:49.226505+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v339: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:49.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:49 smithi078 bash[28259]: cluster 2023-07-29T17:07:49.226505+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v339: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:49.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:49 smithi078 bash[23704]: cluster 2023-07-29T17:07:49.226505+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v339: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:51.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:51 smithi129 bash[23526]: cluster 2023-07-29T17:07:51.228033+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v340: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:51.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:51 smithi078 bash[23704]: cluster 2023-07-29T17:07:51.228033+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v340: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:51.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:51 smithi078 bash[28259]: cluster 2023-07-29T17:07:51.228033+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v340: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:53.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:53 smithi129 bash[23526]: cluster 2023-07-29T17:07:53.229030+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v341: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:53.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:53 smithi078 bash[23704]: cluster 2023-07-29T17:07:53.229030+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v341: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:53.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:53 smithi078 bash[28259]: cluster 2023-07-29T17:07:53.229030+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v341: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:55.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:55 smithi129 bash[23526]: cluster 2023-07-29T17:07:55.230236+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v342: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:55.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:55 smithi078 bash[23704]: cluster 2023-07-29T17:07:55.230236+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v342: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:55.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:55 smithi078 bash[28259]: cluster 2023-07-29T17:07:55.230236+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v342: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:57.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:57 smithi129 bash[23526]: cluster 2023-07-29T17:07:57.231533+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v343: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:57.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:57 smithi078 bash[28259]: cluster 2023-07-29T17:07:57.231533+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v343: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:57.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:57 smithi078 bash[23704]: cluster 2023-07-29T17:07:57.231533+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v343: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:07:59.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:07:59 smithi129 bash[23526]: cluster 2023-07-29T17:07:59.232409+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v344: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:59.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:07:59 smithi078 bash[23704]: cluster 2023-07-29T17:07:59.232409+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v344: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:07:59.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:07:59 smithi078 bash[28259]: cluster 2023-07-29T17:07:59.232409+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v344: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:01.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:01 smithi129 bash[23526]: cluster 2023-07-29T17:08:01.233908+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v345: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:01.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:01 smithi078 bash[23704]: cluster 2023-07-29T17:08:01.233908+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v345: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:01.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:01 smithi078 bash[28259]: cluster 2023-07-29T17:08:01.233908+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v345: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:03.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:03 smithi129 bash[23526]: cluster 2023-07-29T17:08:03.234944+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v346: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:03.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:03 smithi078 bash[23704]: cluster 2023-07-29T17:08:03.234944+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v346: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:03.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:03 smithi078 bash[28259]: cluster 2023-07-29T17:08:03.234944+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v346: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:05.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:05 smithi129 bash[23526]: cluster 2023-07-29T17:08:05.236152+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v347: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:05.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:05 smithi078 bash[23704]: cluster 2023-07-29T17:08:05.236152+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v347: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:05.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:05 smithi078 bash[28259]: cluster 2023-07-29T17:08:05.236152+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v347: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:07.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:07 smithi129 bash[23526]: cluster 2023-07-29T17:08:07.237413+0000 mgr.y (mgr.14144) 375 : cluster [DBG] pgmap v348: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:07.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:07 smithi078 bash[28259]: cluster 2023-07-29T17:08:07.237413+0000 mgr.y (mgr.14144) 375 : cluster [DBG] pgmap v348: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:07.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:07 smithi078 bash[23704]: cluster 2023-07-29T17:08:07.237413+0000 mgr.y (mgr.14144) 375 : cluster [DBG] pgmap v348: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:09.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:09 smithi129 bash[23526]: cluster 2023-07-29T17:08:09.238659+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v349: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:09.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:09 smithi078 bash[23704]: cluster 2023-07-29T17:08:09.238659+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v349: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:09.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:09 smithi078 bash[28259]: cluster 2023-07-29T17:08:09.238659+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v349: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:11.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:11 smithi129 bash[23526]: cluster 2023-07-29T17:08:11.240132+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v350: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:11.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:11 smithi078 bash[28259]: cluster 2023-07-29T17:08:11.240132+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v350: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:11.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:11 smithi078 bash[23704]: cluster 2023-07-29T17:08:11.240132+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v350: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:13.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-07-29T17:08:13.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:13 smithi129 bash[23526]: cluster 2023-07-29T17:08:13.241171+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v351: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:13.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:13 smithi078 bash[28259]: cluster 2023-07-29T17:08:13.241171+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v351: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:13.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:13 smithi078 bash[23704]: cluster 2023-07-29T17:08:13.241171+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v351: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:15.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:15 smithi129 bash[23526]: cluster 2023-07-29T17:08:15.242373+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v352: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:15.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:15 smithi078 bash[23704]: cluster 2023-07-29T17:08:15.242373+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v352: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:15.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:15 smithi078 bash[28259]: cluster 2023-07-29T17:08:15.242373+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v352: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:17.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-07-29T17:08:18.166 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:17 smithi129 bash[23526]: audit 2023-07-29T17:08:16.808975+0000 mon.a (mon.0) 634 : audit [INF] from='client.? 172.21.15.78:0/3098031042' entity='client.admin' 2023-07-29T17:08:18.166 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:17 smithi129 bash[23526]: cluster 2023-07-29T17:08:17.243419+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v353: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:18.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:17 smithi078 bash[23704]: audit 2023-07-29T17:08:16.808975+0000 mon.a (mon.0) 634 : audit [INF] from='client.? 172.21.15.78:0/3098031042' entity='client.admin' 2023-07-29T17:08:18.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:17 smithi078 bash[23704]: cluster 2023-07-29T17:08:17.243419+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v353: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:18.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:17 smithi078 bash[28259]: audit 2023-07-29T17:08:16.808975+0000 mon.a (mon.0) 634 : audit [INF] from='client.? 172.21.15.78:0/3098031042' entity='client.admin' 2023-07-29T17:08:18.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:17 smithi078 bash[28259]: cluster 2023-07-29T17:08:17.243419+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v353: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:19.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:18 smithi078 bash[23704]: audit 2023-07-29T17:08:17.986002+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:19.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:18 smithi078 bash[28259]: audit 2023-07-29T17:08:17.986002+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:19.416 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:18 smithi129 bash[23526]: audit 2023-07-29T17:08:17.986002+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:20.416 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:19 smithi129 bash[23526]: cluster 2023-07-29T17:08:19.244378+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v354: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:20.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:19 smithi078 bash[23704]: cluster 2023-07-29T17:08:19.244378+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v354: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:20.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:19 smithi078 bash[28259]: cluster 2023-07-29T17:08:19.244378+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v354: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:21.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph config set global log_to_journald false --force' 2023-07-29T17:08:22.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:21 smithi078 bash[23704]: audit 2023-07-29T17:08:20.907202+0000 mon.a (mon.0) 636 : audit [INF] from='client.? 172.21.15.78:0/1922092413' entity='client.admin' 2023-07-29T17:08:22.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:21 smithi078 bash[23704]: cluster 2023-07-29T17:08:21.245735+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v355: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:22.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:21 smithi078 bash[28259]: audit 2023-07-29T17:08:20.907202+0000 mon.a (mon.0) 636 : audit [INF] from='client.? 172.21.15.78:0/1922092413' entity='client.admin' 2023-07-29T17:08:22.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:21 smithi078 bash[28259]: cluster 2023-07-29T17:08:21.245735+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v355: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:22.416 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:21 smithi129 bash[23526]: audit 2023-07-29T17:08:20.907202+0000 mon.a (mon.0) 636 : audit [INF] from='client.? 172.21.15.78:0/1922092413' entity='client.admin' 2023-07-29T17:08:22.416 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:21 smithi129 bash[23526]: cluster 2023-07-29T17:08:21.245735+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v355: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:23.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:23 smithi078 bash[28259]: cluster 2023-07-29T17:08:23.246835+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v356: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:23.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:23 smithi078 bash[23704]: cluster 2023-07-29T17:08:23.246835+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v356: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:23.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:23 smithi129 bash[23526]: cluster 2023-07-29T17:08:23.246835+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v356: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:25.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2023-07-29T17:08:26.416 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:26 smithi129 bash[23526]: audit 2023-07-29T17:08:25.026987+0000 mon.a (mon.0) 637 : audit [INF] from='client.? ' entity='client.admin' 2023-07-29T17:08:26.416 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:26 smithi129 bash[23526]: cluster 2023-07-29T17:08:25.247813+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v357: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:26.417 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:26 smithi129 bash[23526]: audit 2023-07-29T17:08:25.279418+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:26.417 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:26 smithi129 bash[23526]: audit 2023-07-29T17:08:25.287431+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:26.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[23704]: audit 2023-07-29T17:08:25.026987+0000 mon.a (mon.0) 637 : audit [INF] from='client.? ' entity='client.admin' 2023-07-29T17:08:26.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[23704]: cluster 2023-07-29T17:08:25.247813+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v357: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:26.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[23704]: audit 2023-07-29T17:08:25.279418+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:26.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[23704]: audit 2023-07-29T17:08:25.287431+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:26.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[28259]: audit 2023-07-29T17:08:25.026987+0000 mon.a (mon.0) 637 : audit [INF] from='client.? ' entity='client.admin' 2023-07-29T17:08:26.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[28259]: cluster 2023-07-29T17:08:25.247813+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v357: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:26.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[28259]: audit 2023-07-29T17:08:25.279418+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:26.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:26 smithi078 bash[28259]: audit 2023-07-29T17:08:25.287431+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:27.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:27 smithi078 bash[23704]: cluster 2023-07-29T17:08:27.249110+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v358: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:27.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:27 smithi078 bash[28259]: cluster 2023-07-29T17:08:27.249110+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v358: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:27.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:27 smithi129 bash[23526]: cluster 2023-07-29T17:08:27.249110+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v358: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:29.238 INFO:teuthology.orchestra.run.smithi078.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.132 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-29T17:08:30.144 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-29T17:08:30.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2023-07-29T17:08:30.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[23704]: audit 2023-07-29T17:08:29.215778+0000 mgr.y (mgr.14144) 386 : audit [DBG] from='client.24743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:30.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[23704]: cephadm 2023-07-29T17:08:29.217191+0000 mgr.y (mgr.14144) 387 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[23704]: audit 2023-07-29T17:08:29.224991+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:30.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[23704]: audit 2023-07-29T17:08:29.231771+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:30.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[23704]: cephadm 2023-07-29T17:08:29.238665+0000 mgr.y (mgr.14144) 388 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[23704]: cluster 2023-07-29T17:08:29.249926+0000 mgr.y (mgr.14144) 389 : cluster [DBG] pgmap v359: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:30.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[28259]: audit 2023-07-29T17:08:29.215778+0000 mgr.y (mgr.14144) 386 : audit [DBG] from='client.24743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:30.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[28259]: cephadm 2023-07-29T17:08:29.217191+0000 mgr.y (mgr.14144) 387 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[28259]: audit 2023-07-29T17:08:29.224991+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:30.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[28259]: audit 2023-07-29T17:08:29.231771+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:30.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[28259]: cephadm 2023-07-29T17:08:29.238665+0000 mgr.y (mgr.14144) 388 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:30 smithi078 bash[28259]: cluster 2023-07-29T17:08:29.249926+0000 mgr.y (mgr.14144) 389 : cluster [DBG] pgmap v359: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:30.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:30 smithi129 bash[23526]: audit 2023-07-29T17:08:29.215778+0000 mgr.y (mgr.14144) 386 : audit [DBG] from='client.24743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:30.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:30 smithi129 bash[23526]: cephadm 2023-07-29T17:08:29.217191+0000 mgr.y (mgr.14144) 387 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:30 smithi129 bash[23526]: audit 2023-07-29T17:08:29.224991+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:30.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:30 smithi129 bash[23526]: audit 2023-07-29T17:08:29.231771+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:30.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:30 smithi129 bash[23526]: cephadm 2023-07-29T17:08:29.238665+0000 mgr.y (mgr.14144) 388 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T17:08:30.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:30 smithi129 bash[23526]: cluster 2023-07-29T17:08:29.249926+0000 mgr.y (mgr.14144) 389 : cluster [DBG] pgmap v359: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:31.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:31 smithi129 bash[23526]: cluster 2023-07-29T17:08:31.250763+0000 mgr.y (mgr.14144) 390 : cluster [DBG] pgmap v360: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:31.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:31 smithi078 bash[23704]: cluster 2023-07-29T17:08:31.250763+0000 mgr.y (mgr.14144) 390 : cluster [DBG] pgmap v360: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:31.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:31 smithi078 bash[28259]: cluster 2023-07-29T17:08:31.250763+0000 mgr.y (mgr.14144) 390 : cluster [DBG] pgmap v360: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:33.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:33 smithi129 bash[23526]: cluster 2023-07-29T17:08:33.251644+0000 mgr.y (mgr.14144) 391 : cluster [DBG] pgmap v361: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:33.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:33 smithi078 bash[23704]: cluster 2023-07-29T17:08:33.251644+0000 mgr.y (mgr.14144) 391 : cluster [DBG] pgmap v361: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:33.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:33 smithi078 bash[28259]: cluster 2023-07-29T17:08:33.251644+0000 mgr.y (mgr.14144) 391 : cluster [DBG] pgmap v361: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:35.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:35 smithi129 bash[23526]: cluster 2023-07-29T17:08:35.252508+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v362: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:35.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:35 smithi129 bash[23526]: audit 2023-07-29T17:08:35.289709+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:08:35.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:35 smithi129 bash[23526]: audit 2023-07-29T17:08:35.292309+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:08:35.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:35 smithi078 bash[23704]: cluster 2023-07-29T17:08:35.252508+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v362: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:35.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:35 smithi078 bash[23704]: audit 2023-07-29T17:08:35.289709+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:08:35.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:35 smithi078 bash[23704]: audit 2023-07-29T17:08:35.292309+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:08:35.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:35 smithi078 bash[28259]: cluster 2023-07-29T17:08:35.252508+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v362: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:35.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:35 smithi078 bash[28259]: audit 2023-07-29T17:08:35.289709+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:08:35.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:35 smithi078 bash[28259]: audit 2023-07-29T17:08:35.292309+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:08:37.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:37 smithi078 bash[23704]: cluster 2023-07-29T17:08:37.253329+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v363: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:37.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:37 smithi078 bash[28259]: cluster 2023-07-29T17:08:37.253329+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v363: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:37.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:37 smithi129 bash[23526]: cluster 2023-07-29T17:08:37.253329+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v363: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:39.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:39 smithi129 bash[23526]: cluster 2023-07-29T17:08:39.254114+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v364: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:39.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:39 smithi078 bash[23704]: cluster 2023-07-29T17:08:39.254114+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v364: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:39.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:39 smithi078 bash[28259]: cluster 2023-07-29T17:08:39.254114+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v364: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:41.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:41 smithi129 bash[23526]: cluster 2023-07-29T17:08:41.255254+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v365: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:41.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:41 smithi078 bash[28259]: cluster 2023-07-29T17:08:41.255254+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v365: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:41.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:41 smithi078 bash[23704]: cluster 2023-07-29T17:08:41.255254+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v365: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:43.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:43 smithi129 bash[23526]: cluster 2023-07-29T17:08:43.256123+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v366: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:43.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:43 smithi078 bash[23704]: cluster 2023-07-29T17:08:43.256123+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v366: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:43 smithi078 bash[28259]: cluster 2023-07-29T17:08:43.256123+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v366: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:45.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:45 smithi129 bash[23526]: cluster 2023-07-29T17:08:45.257250+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v367: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:45.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:45 smithi078 bash[23704]: cluster 2023-07-29T17:08:45.257250+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v367: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:45.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:45 smithi078 bash[28259]: cluster 2023-07-29T17:08:45.257250+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v367: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:47.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:47 smithi078 bash[23704]: cluster 2023-07-29T17:08:47.258507+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v368: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:47.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:47 smithi078 bash[28259]: cluster 2023-07-29T17:08:47.258507+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v368: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:47.916 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:47 smithi129 bash[23526]: cluster 2023-07-29T17:08:47.258507+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v368: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:49.484 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:08:49.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:49 smithi129 bash[23526]: cluster 2023-07-29T17:08:49.259257+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v369: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:49.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:49 smithi078 bash[23704]: cluster 2023-07-29T17:08:49.259257+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v369: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:49.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:49 smithi078 bash[28259]: cluster 2023-07-29T17:08:49.259257+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v369: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:50.169 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2023-07-29T17:08:50.170 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 running (4m) 2m ago 4m *:9093 *:9094 0.20.0 0881eb8f169f 6b73f9bccabd 2023-07-29T17:08:50.170 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 running (4m) 2m ago 4m *:3000 6.7.4 557c83e11646 a2bc911e186a 2023-07-29T17:08:50.170 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (2m) 2m ago 2m - 3.4 24ecd6d5f14c 6685d4a4e408 2023-07-29T17:08:50.170 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 running (9m) 2m ago 9m *:8443 16.2.0 24ecd6d5f14c ccdf793c2cdb 2023-07-29T17:08:50.170 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 running (11m) 2m ago 11m *:9283 16.2.0 24ecd6d5f14c 97596eccd9c9 2023-07-29T17:08:50.170 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (11m) 2m ago 11m - 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (9m) 2m ago 9m - 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (9m) 2m ago 9m - 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 running (5m) 2m ago 5m *:9100 0.18.1 e5a616e4b9cf a6dca72e6f42 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 running (5m) 2m ago 5m *:9100 0.18.1 e5a616e4b9cf 4280de6ba851 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (8m) 2m ago 8m - 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (8m) 2m ago 8m - 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:08:50.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8m) 2m ago 8m - 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (5m) 2m ago 6m - 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 running (4m) 2m ago 5m *:9095 2.18.1 de242295e225 0cfe64e114f3 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 running (3m) 2m ago 3m *:8000 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:08:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 running (2m) 2m ago 2m *:8000 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:08:50.173 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:08:50.173 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:08:50.543 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:08:50.543 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:08:50.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:08:50.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:08:50.543 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:08:50.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2023-07-29T17:08:50.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:08:50.544 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:08:50.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:08:50.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:08:50.544 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:08:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:08:50.890 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:08:50.890 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:08:50.891 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:08:50.891 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2023-07-29T17:08:50.891 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "0/23 ceph daemons upgraded", 2023-07-29T17:08:50.891 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 image on host smithi129" 2023-07-29T17:08:50.891 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:08:51.259 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:08:51.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: audit 2023-07-29T17:08:49.457196+0000 mgr.y (mgr.14144) 400 : audit [DBG] from='client.24749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: audit 2023-07-29T17:08:49.808945+0000 mgr.y (mgr.14144) 401 : audit [DBG] from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: audit 2023-07-29T17:08:50.143603+0000 mgr.y (mgr.14144) 402 : audit [DBG] from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: audit 2023-07-29T17:08:50.174105+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:51.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: cephadm 2023-07-29T17:08:50.174972+0000 mgr.y (mgr.14144) 403 : cephadm [INF] Upgrade: Target is version 18.1.3-11-g0484097b (unknown) 2023-07-29T17:08:51.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: cephadm 2023-07-29T17:08:50.175112+0000 mgr.y (mgr.14144) 404 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dec2dadc08a2f5d927a831d583c48fe98cee7ad6a4f1b5f6644d3d31ec97903f'] 2023-07-29T17:08:51.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: audit 2023-07-29T17:08:50.176981+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:08:51.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: cephadm 2023-07-29T17:08:50.178395+0000 mgr.y (mgr.14144) 405 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-29T17:08:51.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:51 smithi129 bash[23526]: audit 2023-07-29T17:08:50.529306+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? 172.21.15.78:0/1708062888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:08:51.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: audit 2023-07-29T17:08:49.457196+0000 mgr.y (mgr.14144) 400 : audit [DBG] from='client.24749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: audit 2023-07-29T17:08:49.808945+0000 mgr.y (mgr.14144) 401 : audit [DBG] from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: audit 2023-07-29T17:08:50.143603+0000 mgr.y (mgr.14144) 402 : audit [DBG] from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: audit 2023-07-29T17:08:50.174105+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: cephadm 2023-07-29T17:08:50.174972+0000 mgr.y (mgr.14144) 403 : cephadm [INF] Upgrade: Target is version 18.1.3-11-g0484097b (unknown) 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: cephadm 2023-07-29T17:08:50.175112+0000 mgr.y (mgr.14144) 404 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dec2dadc08a2f5d927a831d583c48fe98cee7ad6a4f1b5f6644d3d31ec97903f'] 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: audit 2023-07-29T17:08:50.176981+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: cephadm 2023-07-29T17:08:50.178395+0000 mgr.y (mgr.14144) 405 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-29T17:08:51.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[23704]: audit 2023-07-29T17:08:50.529306+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? 172.21.15.78:0/1708062888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: audit 2023-07-29T17:08:49.457196+0000 mgr.y (mgr.14144) 400 : audit [DBG] from='client.24749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: audit 2023-07-29T17:08:49.808945+0000 mgr.y (mgr.14144) 401 : audit [DBG] from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: audit 2023-07-29T17:08:50.143603+0000 mgr.y (mgr.14144) 402 : audit [DBG] from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: audit 2023-07-29T17:08:50.174105+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: cephadm 2023-07-29T17:08:50.174972+0000 mgr.y (mgr.14144) 403 : cephadm [INF] Upgrade: Target is version 18.1.3-11-g0484097b (unknown) 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: cephadm 2023-07-29T17:08:50.175112+0000 mgr.y (mgr.14144) 404 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dec2dadc08a2f5d927a831d583c48fe98cee7ad6a4f1b5f6644d3d31ec97903f'] 2023-07-29T17:08:51.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: audit 2023-07-29T17:08:50.176981+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:08:51.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: cephadm 2023-07-29T17:08:50.178395+0000 mgr.y (mgr.14144) 405 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-29T17:08:51.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:51 smithi078 bash[28259]: audit 2023-07-29T17:08:50.529306+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? 172.21.15.78:0/1708062888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:08:52.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:52 smithi129 bash[23526]: cephadm 2023-07-29T17:08:50.394601+0000 mgr.y (mgr.14144) 406 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 on smithi129 2023-07-29T17:08:52.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:52 smithi129 bash[23526]: audit 2023-07-29T17:08:50.877097+0000 mgr.y (mgr.14144) 407 : audit [DBG] from='client.14859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:52.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:52 smithi129 bash[23526]: audit 2023-07-29T17:08:51.246072+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.78:0/1279175204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:08:52.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:52 smithi078 bash[23704]: cephadm 2023-07-29T17:08:50.394601+0000 mgr.y (mgr.14144) 406 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 on smithi129 2023-07-29T17:08:52.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:52 smithi078 bash[23704]: audit 2023-07-29T17:08:50.877097+0000 mgr.y (mgr.14144) 407 : audit [DBG] from='client.14859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:52.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:52 smithi078 bash[23704]: audit 2023-07-29T17:08:51.246072+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.78:0/1279175204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:08:52.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:52 smithi078 bash[28259]: cephadm 2023-07-29T17:08:50.394601+0000 mgr.y (mgr.14144) 406 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 on smithi129 2023-07-29T17:08:52.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:52 smithi078 bash[28259]: audit 2023-07-29T17:08:50.877097+0000 mgr.y (mgr.14144) 407 : audit [DBG] from='client.14859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:08:52.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:52 smithi078 bash[28259]: audit 2023-07-29T17:08:51.246072+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.78:0/1279175204' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:08:53.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:53 smithi129 bash[23526]: cluster 2023-07-29T17:08:51.260630+0000 mgr.y (mgr.14144) 408 : cluster [DBG] pgmap v370: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:53.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:53 smithi078 bash[23704]: cluster 2023-07-29T17:08:51.260630+0000 mgr.y (mgr.14144) 408 : cluster [DBG] pgmap v370: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:53.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:53 smithi078 bash[28259]: cluster 2023-07-29T17:08:51.260630+0000 mgr.y (mgr.14144) 408 : cluster [DBG] pgmap v370: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:55.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:55 smithi129 bash[23526]: cluster 2023-07-29T17:08:53.261649+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v371: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:55.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:55 smithi078 bash[23704]: cluster 2023-07-29T17:08:53.261649+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v371: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:55.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:55 smithi078 bash[28259]: cluster 2023-07-29T17:08:53.261649+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v371: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:57.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:57 smithi129 bash[23526]: cluster 2023-07-29T17:08:55.262812+0000 mgr.y (mgr.14144) 410 : cluster [DBG] pgmap v372: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:57.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:57 smithi078 bash[23704]: cluster 2023-07-29T17:08:55.262812+0000 mgr.y (mgr.14144) 410 : cluster [DBG] pgmap v372: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:57.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:57 smithi078 bash[28259]: cluster 2023-07-29T17:08:55.262812+0000 mgr.y (mgr.14144) 410 : cluster [DBG] pgmap v372: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:08:59.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:08:59 smithi129 bash[23526]: cluster 2023-07-29T17:08:57.264184+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v373: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:59.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:08:59 smithi078 bash[23704]: cluster 2023-07-29T17:08:57.264184+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v373: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:08:59.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:08:59 smithi078 bash[28259]: cluster 2023-07-29T17:08:57.264184+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v373: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:01.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:01 smithi129 bash[23526]: cluster 2023-07-29T17:08:59.265186+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v374: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:01.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:01 smithi078 bash[23704]: cluster 2023-07-29T17:08:59.265186+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v374: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:01.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:01 smithi078 bash[28259]: cluster 2023-07-29T17:08:59.265186+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v374: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:03.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:03 smithi129 bash[23526]: cluster 2023-07-29T17:09:01.266546+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v375: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:03.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:03 smithi078 bash[28259]: cluster 2023-07-29T17:09:01.266546+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v375: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:03.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:03 smithi078 bash[23704]: cluster 2023-07-29T17:09:01.266546+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v375: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:05.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:05 smithi129 bash[23526]: cluster 2023-07-29T17:09:03.267584+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v376: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:05.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:05 smithi078 bash[28259]: cluster 2023-07-29T17:09:03.267584+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v376: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:05.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:05 smithi078 bash[23704]: cluster 2023-07-29T17:09:03.267584+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v376: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:07.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:07 smithi129 bash[23526]: cluster 2023-07-29T17:09:05.268696+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v377: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:07.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:07 smithi078 bash[28259]: cluster 2023-07-29T17:09:05.268696+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v377: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:07.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:07 smithi078 bash[23704]: cluster 2023-07-29T17:09:05.268696+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v377: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:09.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:09 smithi129 bash[23526]: cluster 2023-07-29T17:09:07.269996+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v378: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:09.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:09 smithi078 bash[28259]: cluster 2023-07-29T17:09:07.269996+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v378: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:09.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:09 smithi078 bash[23704]: cluster 2023-07-29T17:09:07.269996+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v378: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:11.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:11 smithi129 bash[23526]: cluster 2023-07-29T17:09:09.270957+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v379: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:11.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:11 smithi078 bash[23704]: cluster 2023-07-29T17:09:09.270957+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v379: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:11.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:11 smithi078 bash[28259]: cluster 2023-07-29T17:09:09.270957+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v379: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:13.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:13 smithi129 bash[23526]: cluster 2023-07-29T17:09:11.272359+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v380: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:13.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:13 smithi078 bash[23704]: cluster 2023-07-29T17:09:11.272359+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v380: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:13.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:13 smithi078 bash[28259]: cluster 2023-07-29T17:09:11.272359+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v380: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:15.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:15 smithi129 bash[23526]: cluster 2023-07-29T17:09:13.273319+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v381: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:15.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:15 smithi078 bash[23704]: cluster 2023-07-29T17:09:13.273319+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v381: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:15.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:15 smithi078 bash[28259]: cluster 2023-07-29T17:09:13.273319+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v381: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:17.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:17 smithi129 bash[23526]: cluster 2023-07-29T17:09:15.274543+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v382: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:17.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:17 smithi129 bash[23526]: audit 2023-07-29T17:09:17.164817+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:17.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:17 smithi129 bash[23526]: audit 2023-07-29T17:09:17.166958+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:09:17.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:17 smithi129 bash[23526]: audit 2023-07-29T17:09:17.168751+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:09:17.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:17 smithi129 bash[23526]: audit 2023-07-29T17:09:17.170149+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:09:17.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[23704]: cluster 2023-07-29T17:09:15.274543+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v382: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:17.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[23704]: audit 2023-07-29T17:09:17.164817+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:17.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[23704]: audit 2023-07-29T17:09:17.166958+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:09:17.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[23704]: audit 2023-07-29T17:09:17.168751+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:09:17.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[23704]: audit 2023-07-29T17:09:17.170149+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:09:17.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[28259]: cluster 2023-07-29T17:09:15.274543+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v382: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:17.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[28259]: audit 2023-07-29T17:09:17.164817+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:17.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[28259]: audit 2023-07-29T17:09:17.166958+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:09:17.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[28259]: audit 2023-07-29T17:09:17.168751+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:09:17.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:17 smithi078 bash[28259]: audit 2023-07-29T17:09:17.170149+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:09:18.617 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:18 smithi129 bash[23526]: cephadm 2023-07-29T17:09:17.155570+0000 mgr.y (mgr.14144) 421 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-29T17:09:18.618 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:18 smithi129 bash[23526]: cephadm 2023-07-29T17:09:17.171556+0000 mgr.y (mgr.14144) 422 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T17:09:18.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:18 smithi078 bash[23704]: cephadm 2023-07-29T17:09:17.155570+0000 mgr.y (mgr.14144) 421 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-29T17:09:18.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:18 smithi078 bash[23704]: cephadm 2023-07-29T17:09:17.171556+0000 mgr.y (mgr.14144) 422 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T17:09:18.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:18 smithi078 bash[28259]: cephadm 2023-07-29T17:09:17.155570+0000 mgr.y (mgr.14144) 421 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-29T17:09:18.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:18 smithi078 bash[28259]: cephadm 2023-07-29T17:09:17.171556+0000 mgr.y (mgr.14144) 422 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T17:09:19.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:19 smithi129 bash[23526]: cluster 2023-07-29T17:09:17.275783+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v383: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:19.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:19 smithi078 bash[23704]: cluster 2023-07-29T17:09:17.275783+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v383: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:19.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:19 smithi078 bash[28259]: cluster 2023-07-29T17:09:17.275783+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v383: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:19.883 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:19 smithi129 systemd[1]: Stopping Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:09:20.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:20 smithi129 docker[43015]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mgr.x 2023-07-29T17:09:20.654 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:20 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:09:20.654 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:20 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2023-07-29T17:09:20.654 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:20 smithi129 systemd[1]: Stopped Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:09:20.654 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:20 smithi129 systemd[1]: Started Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:09:21.626 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:09:21.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:21 smithi129 bash[23526]: cluster 2023-07-29T17:09:19.276901+0000 mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v384: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:21.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:21 smithi129 bash[23526]: audit 2023-07-29T17:09:20.684903+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:21.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:21 smithi129 bash[23526]: audit 2023-07-29T17:09:20.936543+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:21.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:21 smithi078 bash[28259]: cluster 2023-07-29T17:09:19.276901+0000 mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v384: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:21.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:21 smithi078 bash[28259]: audit 2023-07-29T17:09:20.684903+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:21.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:21 smithi078 bash[28259]: audit 2023-07-29T17:09:20.936543+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:21.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:21 smithi078 bash[23704]: cluster 2023-07-29T17:09:19.276901+0000 mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v384: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:21.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:21 smithi078 bash[23704]: audit 2023-07-29T17:09:20.684903+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:21.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:21 smithi078 bash[23704]: audit 2023-07-29T17:09:20.936543+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:22.067 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:21 smithi129 bash[43116]: debug 2023-07-29T17:09:21.887+0000 7fd24aa7a700 1 -- 172.21.15.129:0/3204636008 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5559ef2491e0 con 0x5559effc8800 2023-07-29T17:09:22.332 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2023-07-29T17:09:22.332 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 running (4m) 3m ago 5m *:9093 *:9094 0.20.0 0881eb8f169f 6b73f9bccabd 2023-07-29T17:09:22.332 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 running (5m) 3m ago 5m *:3000 6.7.4 557c83e11646 a2bc911e186a 2023-07-29T17:09:22.332 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (3m) 3m ago 3m - 3.4 24ecd6d5f14c 6685d4a4e408 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 starting - - *:8443 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 running (11m) 3m ago 11m *:9283 16.2.0 24ecd6d5f14c 97596eccd9c9 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (11m) 3m ago 11m - 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (10m) 3m ago 10m - 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 3m ago 10m - 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 running (5m) 3m ago 5m *:9100 0.18.1 e5a616e4b9cf a6dca72e6f42 2023-07-29T17:09:22.333 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 running (5m) 3m ago 5m *:9100 0.18.1 e5a616e4b9cf 4280de6ba851 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (9m) 3m ago 9m - 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (8m) 3m ago 9m - 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8m) 3m ago 8m - 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (8m) 3m ago 8m - 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (7m) 3m ago 7m - 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (7m) 3m ago 7m - 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (6m) 3m ago 6m - 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:09:22.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (6m) 3m ago 6m - 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:09:22.335 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 running (4m) 3m ago 5m *:9095 2.18.1 de242295e225 0cfe64e114f3 2023-07-29T17:09:22.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 running (3m) 3m ago 3m *:8000 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:09:22.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 running (3m) 3m ago 3m *:8000 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:09:22.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 running (3m) 3m ago 3m *:80 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:09:22.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 running (3m) 3m ago 3m *:80 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:09:22.417 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:22 smithi129 bash[43116]: debug 2023-07-29T17:09:22.047+0000 7fd254bc8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T17:09:22.707 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:09:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:09:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:09:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:22.707 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:09:22.708 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:09:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:09:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:09:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2023-07-29T17:09:22.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:09:22.709 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:09:23.053 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:09:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:09:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:09:23.053 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2023-07-29T17:09:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "", 2023-07-29T17:09:23.054 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons" 2023-07-29T17:09:23.054 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:09:23.283 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:22 smithi129 bash[43116]: debug 2023-07-29T17:09:22.927+0000 7fd254bc8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T17:09:23.283 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[43116]: debug 2023-07-29T17:09:23.027+0000 7fd254bc8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T17:09:23.284 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[43116]: debug 2023-07-29T17:09:23.123+0000 7fd254bc8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T17:09:23.435 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:09:23.667 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[43116]: debug 2023-07-29T17:09:23.387+0000 7fd254bc8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:09:23.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[23526]: cluster 2023-07-29T17:09:21.277689+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v385: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:23.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[23526]: audit 2023-07-29T17:09:21.598169+0000 mgr.y (mgr.14144) 426 : audit [DBG] from='client.24785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:23.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[23526]: audit 2023-07-29T17:09:21.949620+0000 mgr.y (mgr.14144) 427 : audit [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:23.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[23526]: audit 2023-07-29T17:09:22.692497+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? 172.21.15.78:0/4141059741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:09:23.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[23704]: cluster 2023-07-29T17:09:21.277689+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v385: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:23.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[23704]: audit 2023-07-29T17:09:21.598169+0000 mgr.y (mgr.14144) 426 : audit [DBG] from='client.24785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:23.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[23704]: audit 2023-07-29T17:09:21.949620+0000 mgr.y (mgr.14144) 427 : audit [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:23.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[23704]: audit 2023-07-29T17:09:22.692497+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? 172.21.15.78:0/4141059741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:09:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[28259]: cluster 2023-07-29T17:09:21.277689+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v385: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[28259]: audit 2023-07-29T17:09:21.598169+0000 mgr.y (mgr.14144) 426 : audit [DBG] from='client.24785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[28259]: audit 2023-07-29T17:09:21.949620+0000 mgr.y (mgr.14144) 427 : audit [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:23 smithi078 bash[28259]: audit 2023-07-29T17:09:22.692497+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? 172.21.15.78:0/4141059741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:09:24.167 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[43116]: debug 2023-07-29T17:09:23.735+0000 7fd254bc8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T17:09:24.167 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:23 smithi129 bash[43116]: debug 2023-07-29T17:09:23.927+0000 7fd254bc8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T17:09:24.667 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:24 smithi129 bash[43116]: debug 2023-07-29T17:09:24.163+0000 7fd254bc8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T17:09:24.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:24 smithi129 bash[23526]: audit 2023-07-29T17:09:22.295931+0000 mgr.y (mgr.14144) 428 : audit [DBG] from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:24.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:24 smithi129 bash[23526]: audit 2023-07-29T17:09:23.038700+0000 mgr.y (mgr.14144) 429 : audit [DBG] from='client.14901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:24.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:24 smithi129 bash[23526]: cluster 2023-07-29T17:09:23.278221+0000 mgr.y (mgr.14144) 430 : cluster [DBG] pgmap v386: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:24.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:24 smithi129 bash[23526]: audit 2023-07-29T17:09:23.421956+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.78:0/3181215766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:09:24.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[23704]: audit 2023-07-29T17:09:22.295931+0000 mgr.y (mgr.14144) 428 : audit [DBG] from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:24.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[23704]: audit 2023-07-29T17:09:23.038700+0000 mgr.y (mgr.14144) 429 : audit [DBG] from='client.14901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:24.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[23704]: cluster 2023-07-29T17:09:23.278221+0000 mgr.y (mgr.14144) 430 : cluster [DBG] pgmap v386: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:24.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[23704]: audit 2023-07-29T17:09:23.421956+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.78:0/3181215766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:09:24.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[28259]: audit 2023-07-29T17:09:22.295931+0000 mgr.y (mgr.14144) 428 : audit [DBG] from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:24.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[28259]: audit 2023-07-29T17:09:23.038700+0000 mgr.y (mgr.14144) 429 : audit [DBG] from='client.14901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:24.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[28259]: cluster 2023-07-29T17:09:23.278221+0000 mgr.y (mgr.14144) 430 : cluster [DBG] pgmap v386: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:09:24.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:24 smithi078 bash[28259]: audit 2023-07-29T17:09:23.421956+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.78:0/3181215766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:09:25.524 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:25 smithi129 bash[43116]: debug 2023-07-29T17:09:25.255+0000 7fd254bc8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T17:09:25.524 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:25 smithi129 bash[43116]: debug 2023-07-29T17:09:25.379+0000 7fd254bc8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T17:09:25.917 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:25 smithi129 bash[43116]: debug 2023-07-29T17:09:25.511+0000 7fd254bc8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T17:09:26.313 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:25 smithi129 bash[43116]: debug 2023-07-29T17:09:25.995+0000 7fd254bc8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T17:09:26.314 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[43116]: debug 2023-07-29T17:09:26.219+0000 7fd254bc8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T17:09:26.610 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: cluster 2023-07-29T17:09:25.279324+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v387: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:26.610 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: audit 2023-07-29T17:09:25.301744+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:26.610 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: audit 2023-07-29T17:09:25.559016+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:26.610 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: audit 2023-07-29T17:09:25.568589+0000 mon.a (mon.0) 656 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:09:26.610 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: cephadm 2023-07-29T17:09:25.570433+0000 mgr.y (mgr.14144) 432 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-29T17:09:26.611 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: cephadm 2023-07-29T17:09:25.573612+0000 mgr.y (mgr.14144) 433 : cephadm [INF] Failing over to other MGR 2023-07-29T17:09:26.611 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: audit 2023-07-29T17:09:25.574118+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-07-29T17:09:26.611 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[23526]: cluster 2023-07-29T17:09:25.584724+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-07-29T17:09:26.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: cluster 2023-07-29T17:09:25.279324+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v387: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:26.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: audit 2023-07-29T17:09:25.301744+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:26.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: audit 2023-07-29T17:09:25.559016+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:26.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: audit 2023-07-29T17:09:25.568589+0000 mon.a (mon.0) 656 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:09:26.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: cephadm 2023-07-29T17:09:25.570433+0000 mgr.y (mgr.14144) 432 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-29T17:09:26.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: cephadm 2023-07-29T17:09:25.573612+0000 mgr.y (mgr.14144) 433 : cephadm [INF] Failing over to other MGR 2023-07-29T17:09:26.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: audit 2023-07-29T17:09:25.574118+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-07-29T17:09:26.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[28259]: cluster 2023-07-29T17:09:25.584724+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-07-29T17:09:26.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: cluster 2023-07-29T17:09:25.279324+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v387: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:09:26.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: audit 2023-07-29T17:09:25.301744+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:26.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: audit 2023-07-29T17:09:25.559016+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' 2023-07-29T17:09:26.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: audit 2023-07-29T17:09:25.568589+0000 mon.a (mon.0) 656 : audit [DBG] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:09:26.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: cephadm 2023-07-29T17:09:25.570433+0000 mgr.y (mgr.14144) 432 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-29T17:09:26.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: cephadm 2023-07-29T17:09:25.573612+0000 mgr.y (mgr.14144) 433 : cephadm [INF] Failing over to other MGR 2023-07-29T17:09:26.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: audit 2023-07-29T17:09:25.574118+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-07-29T17:09:26.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23704]: cluster 2023-07-29T17:09:25.584724+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-07-29T17:09:26.917 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:26 smithi129 bash[43116]: debug 2023-07-29T17:09:26.595+0000 7fd254bc8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T17:09:26.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23973]: ignoring --setuser ceph since I am not root 2023-07-29T17:09:26.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23973]: ignoring --setgroup ceph since I am not root 2023-07-29T17:09:26.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:09:26 smithi078 bash[23973]: debug 2023-07-29T17:09:26.610+0000 7f5f88b12700 1 -- 172.21.15.78:0/2299264956 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5581cc9e04e0 con 0x5581ccad1000 2023-07-29T17:09:27.418 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:27 smithi129 bash[43116]: debug 2023-07-29T17:09:27.083+0000 7fd254bc8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:09:27.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:27 smithi129 bash[23526]: audit 2023-07-29T17:09:26.565981+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-07-29T17:09:27.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:27 smithi129 bash[23526]: cluster 2023-07-29T17:09:26.566167+0000 mon.a (mon.0) 660 : cluster [DBG] mgrmap e16: x(active, starting, since 0.990936s) 2023-07-29T17:09:27.918 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:27 smithi129 bash[43116]: debug 2023-07-29T17:09:27.631+0000 7fd254bc8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T17:09:27.918 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:27 smithi129 bash[43116]: debug 2023-07-29T17:09:27.739+0000 7fd254bc8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T17:09:27.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:27 smithi078 bash[28259]: audit 2023-07-29T17:09:26.565981+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-07-29T17:09:27.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:27 smithi078 bash[28259]: cluster 2023-07-29T17:09:26.566167+0000 mon.a (mon.0) 660 : cluster [DBG] mgrmap e16: x(active, starting, since 0.990936s) 2023-07-29T17:09:27.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:27 smithi078 bash[23704]: audit 2023-07-29T17:09:26.565981+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14144 172.21.15.78:0/1145233953' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-07-29T17:09:27.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:27 smithi078 bash[23704]: cluster 2023-07-29T17:09:26.566167+0000 mon.a (mon.0) 660 : cluster [DBG] mgrmap e16: x(active, starting, since 0.990936s) 2023-07-29T17:09:28.353 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.083+0000 7fd254bc8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T17:09:28.354 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.199+0000 7fd254bc8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T17:09:28.667 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.339+0000 7fd254bc8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T17:09:28.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.451+0000 7fd254bc8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T17:09:28.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.575+0000 7fd254bc8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T17:09:29.167 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.763+0000 7fd254bc8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T17:09:29.168 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:28 smithi129 bash[43116]: debug 2023-07-29T17:09:28.887+0000 7fd254bc8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T17:09:30.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: cluster 2023-07-29T17:09:30.126026+0000 mon.a (mon.0) 661 : cluster [INF] Active manager daemon x restarted 2023-07-29T17:09:30.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: cluster 2023-07-29T17:09:30.127143+0000 mon.a (mon.0) 662 : cluster [INF] Activating manager daemon x 2023-07-29T17:09:30.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: audit 2023-07-29T17:09:30.129210+0000 mon.b (mon.1) 26 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:09:30.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: cluster 2023-07-29T17:09:30.133883+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-07-29T17:09:30.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: audit 2023-07-29T17:09:30.136206+0000 mon.b (mon.1) 27 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:09:30.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: audit 2023-07-29T17:09:30.138066+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:09:30.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[23526]: audit 2023-07-29T17:09:30.138783+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:09:30.422 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:09:30 smithi129 bash[43116]: debug 2023-07-29T17:09:30.102+0000 7fd254bc8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T17:09:30.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: cluster 2023-07-29T17:09:30.126026+0000 mon.a (mon.0) 661 : cluster [INF] Active manager daemon x restarted 2023-07-29T17:09:30.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: cluster 2023-07-29T17:09:30.127143+0000 mon.a (mon.0) 662 : cluster [INF] Activating manager daemon x 2023-07-29T17:09:30.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: audit 2023-07-29T17:09:30.129210+0000 mon.b (mon.1) 26 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:09:30.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: cluster 2023-07-29T17:09:30.133883+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-07-29T17:09:30.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: audit 2023-07-29T17:09:30.136206+0000 mon.b (mon.1) 27 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:09:30.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: audit 2023-07-29T17:09:30.138066+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:09:30.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[28259]: audit 2023-07-29T17:09:30.138783+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:09:30.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: cluster 2023-07-29T17:09:30.126026+0000 mon.a (mon.0) 661 : cluster [INF] Active manager daemon x restarted 2023-07-29T17:09:30.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: cluster 2023-07-29T17:09:30.127143+0000 mon.a (mon.0) 662 : cluster [INF] Activating manager daemon x 2023-07-29T17:09:30.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: audit 2023-07-29T17:09:30.129210+0000 mon.b (mon.1) 26 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:09:30.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: cluster 2023-07-29T17:09:30.133883+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-07-29T17:09:30.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: audit 2023-07-29T17:09:30.136206+0000 mon.b (mon.1) 27 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:09:30.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: audit 2023-07-29T17:09:30.138066+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:09:30.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:30 smithi078 bash[23704]: audit 2023-07-29T17:09:30.138783+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:09:31.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cluster 2023-07-29T17:09:30.184713+0000 mon.a (mon.0) 664 : cluster [DBG] mgrmap e17: x(active, starting, since 0.0577492s) 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.195324+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.195627+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.195886+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.196696+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.196996+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.197367+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:09:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.197710+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.198064+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.198399+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.198778+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.199151+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.199498+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.200736+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.201008+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:09:31.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.201632+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cluster 2023-07-29T17:09:30.299971+0000 mon.a (mon.0) 665 : cluster [INF] Manager daemon x is now available 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cephadm 2023-07-29T17:09:30.324506+0000 mgr.x (mgr.24745) 1 : cephadm [INF] inventory: adjusted host smithi078 addr 'smithi078' -> '172.21.15.78' 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.330314+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.352146+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cephadm 2023-07-29T17:09:30.353973+0000 mgr.x (mgr.24745) 2 : cephadm [INF] Queued rgw.foo for migration 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.359801+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cephadm 2023-07-29T17:09:30.361851+0000 mgr.x (mgr.24745) 3 : cephadm [INF] Queued rgw.smpl for migration 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.369498+0000 mon.b (mon.1) 45 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.397208+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.397483+0000 mon.b (mon.1) 46 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.431696+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.432167+0000 mon.b (mon.1) 47 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.450989+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.469826+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T17:09:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:30.470401+0000 mon.b (mon.1) 48 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T17:09:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cephadm 2023-07-29T17:09:30.833934+0000 mgr.x (mgr.24745) 4 : cephadm [INF] Deploying cephadm binary to smithi129 2023-07-29T17:09:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: cephadm 2023-07-29T17:09:30.837724+0000 mgr.x (mgr.24745) 5 : cephadm [INF] Deploying cephadm binary to smithi078 2023-07-29T17:09:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:31.117926+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:31 smithi129 bash[23526]: audit 2023-07-29T17:09:31.147037+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cluster 2023-07-29T17:09:30.184713+0000 mon.a (mon.0) 664 : cluster [DBG] mgrmap e17: x(active, starting, since 0.0577492s) 2023-07-29T17:09:31.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.195324+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:31.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.195627+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:31.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.195886+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.196696+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.196996+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.197367+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.197710+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.198064+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.198399+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.198778+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:09:31.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.199151+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.199498+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.200736+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.201008+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.201632+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cluster 2023-07-29T17:09:30.299971+0000 mon.a (mon.0) 665 : cluster [INF] Manager daemon x is now available 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cephadm 2023-07-29T17:09:30.324506+0000 mgr.x (mgr.24745) 1 : cephadm [INF] inventory: adjusted host smithi078 addr 'smithi078' -> '172.21.15.78' 2023-07-29T17:09:31.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.330314+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.352146+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cluster 2023-07-29T17:09:30.184713+0000 mon.a (mon.0) 664 : cluster [DBG] mgrmap e17: x(active, starting, since 0.0577492s) 2023-07-29T17:09:31.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.195324+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:31.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.195627+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:31.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.195886+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.196696+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.196996+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.197367+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.197710+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.198064+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.198399+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.198778+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:09:31.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.199151+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.199498+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.200736+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.201008+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.201632+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cluster 2023-07-29T17:09:30.299971+0000 mon.a (mon.0) 665 : cluster [INF] Manager daemon x is now available 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cephadm 2023-07-29T17:09:30.324506+0000 mgr.x (mgr.24745) 1 : cephadm [INF] inventory: adjusted host smithi078 addr 'smithi078' -> '172.21.15.78' 2023-07-29T17:09:31.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.330314+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.352146+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cephadm 2023-07-29T17:09:30.353973+0000 mgr.x (mgr.24745) 2 : cephadm [INF] Queued rgw.foo for migration 2023-07-29T17:09:31.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.359801+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cephadm 2023-07-29T17:09:30.361851+0000 mgr.x (mgr.24745) 3 : cephadm [INF] Queued rgw.smpl for migration 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.369498+0000 mon.b (mon.1) 45 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.397208+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.397483+0000 mon.b (mon.1) 46 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.431696+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.432167+0000 mon.b (mon.1) 47 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.450989+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.469826+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T17:09:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:30.470401+0000 mon.b (mon.1) 48 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T17:09:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cephadm 2023-07-29T17:09:30.833934+0000 mgr.x (mgr.24745) 4 : cephadm [INF] Deploying cephadm binary to smithi129 2023-07-29T17:09:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: cephadm 2023-07-29T17:09:30.837724+0000 mgr.x (mgr.24745) 5 : cephadm [INF] Deploying cephadm binary to smithi078 2023-07-29T17:09:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:31.117926+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[23704]: audit 2023-07-29T17:09:31.147037+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cephadm 2023-07-29T17:09:30.353973+0000 mgr.x (mgr.24745) 2 : cephadm [INF] Queued rgw.foo for migration 2023-07-29T17:09:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.359801+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cephadm 2023-07-29T17:09:30.361851+0000 mgr.x (mgr.24745) 3 : cephadm [INF] Queued rgw.smpl for migration 2023-07-29T17:09:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.369498+0000 mon.b (mon.1) 45 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:09:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.397208+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.397483+0000 mon.b (mon.1) 46 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.431696+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.432167+0000 mon.b (mon.1) 47 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.450989+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.469826+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:30.470401+0000 mon.b (mon.1) 48 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T17:09:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cephadm 2023-07-29T17:09:30.833934+0000 mgr.x (mgr.24745) 4 : cephadm [INF] Deploying cephadm binary to smithi129 2023-07-29T17:09:31.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: cephadm 2023-07-29T17:09:30.837724+0000 mgr.x (mgr.24745) 5 : cephadm [INF] Deploying cephadm binary to smithi078 2023-07-29T17:09:31.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:31.117926+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:31.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:31 smithi078 bash[28259]: audit 2023-07-29T17:09:31.147037+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:32.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cluster 2023-07-29T17:09:31.207890+0000 mon.a (mon.0) 675 : cluster [DBG] mgrmap e18: x(active, since 1.08094s) 2023-07-29T17:09:32.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cluster 2023-07-29T17:09:31.242347+0000 mgr.x (mgr.24745) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:32.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: audit 2023-07-29T17:09:31.361659+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T17:09:32.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cluster 2023-07-29T17:09:31.361855+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-07-29T17:09:32.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: audit 2023-07-29T17:09:31.375011+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:32.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: audit 2023-07-29T17:09:31.375603+0000 mon.b (mon.1) 49 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:32.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cephadm 2023-07-29T17:09:31.607539+0000 mgr.x (mgr.24745) 7 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Bus STARTING 2023-07-29T17:09:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cephadm 2023-07-29T17:09:31.713072+0000 mgr.x (mgr.24745) 8 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Serving on http://172.21.15.129:8765 2023-07-29T17:09:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cephadm 2023-07-29T17:09:31.893675+0000 mgr.x (mgr.24745) 9 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Serving on https://172.21.15.129:7150 2023-07-29T17:09:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:32 smithi129 bash[23526]: cephadm 2023-07-29T17:09:31.894496+0000 mgr.x (mgr.24745) 10 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Bus STARTED 2023-07-29T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cluster 2023-07-29T17:09:31.207890+0000 mon.a (mon.0) 675 : cluster [DBG] mgrmap e18: x(active, since 1.08094s) 2023-07-29T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cluster 2023-07-29T17:09:31.242347+0000 mgr.x (mgr.24745) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: audit 2023-07-29T17:09:31.361659+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cluster 2023-07-29T17:09:31.361855+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-07-29T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: audit 2023-07-29T17:09:31.375011+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: audit 2023-07-29T17:09:31.375603+0000 mon.b (mon.1) 49 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cephadm 2023-07-29T17:09:31.607539+0000 mgr.x (mgr.24745) 7 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Bus STARTING 2023-07-29T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cephadm 2023-07-29T17:09:31.713072+0000 mgr.x (mgr.24745) 8 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Serving on http://172.21.15.129:8765 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cephadm 2023-07-29T17:09:31.893675+0000 mgr.x (mgr.24745) 9 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Serving on https://172.21.15.129:7150 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[23704]: cephadm 2023-07-29T17:09:31.894496+0000 mgr.x (mgr.24745) 10 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Bus STARTED 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cluster 2023-07-29T17:09:31.207890+0000 mon.a (mon.0) 675 : cluster [DBG] mgrmap e18: x(active, since 1.08094s) 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cluster 2023-07-29T17:09:31.242347+0000 mgr.x (mgr.24745) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: audit 2023-07-29T17:09:31.361659+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cluster 2023-07-29T17:09:31.361855+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-07-29T17:09:32.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: audit 2023-07-29T17:09:31.375011+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:32.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: audit 2023-07-29T17:09:31.375603+0000 mon.b (mon.1) 49 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T17:09:32.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cephadm 2023-07-29T17:09:31.607539+0000 mgr.x (mgr.24745) 7 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Bus STARTING 2023-07-29T17:09:32.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cephadm 2023-07-29T17:09:31.713072+0000 mgr.x (mgr.24745) 8 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Serving on http://172.21.15.129:8765 2023-07-29T17:09:32.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cephadm 2023-07-29T17:09:31.893675+0000 mgr.x (mgr.24745) 9 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Serving on https://172.21.15.129:7150 2023-07-29T17:09:32.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:32 smithi078 bash[28259]: cephadm 2023-07-29T17:09:31.894496+0000 mgr.x (mgr.24745) 10 : cephadm [INF] [29/Jul/2023:17:09:31] ENGINE Bus STARTED 2023-07-29T17:09:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: cluster 2023-07-29T17:09:32.201137+0000 mgr.x (mgr.24745) 11 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: audit 2023-07-29T17:09:32.365662+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T17:09:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: cluster 2023-07-29T17:09:32.366022+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-29T17:09:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: cluster 2023-07-29T17:09:32.498096+0000 mon.a (mon.0) 681 : cluster [DBG] Standby manager daemon y started 2023-07-29T17:09:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: audit 2023-07-29T17:09:32.498858+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T17:09:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: audit 2023-07-29T17:09:32.503782+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:09:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: audit 2023-07-29T17:09:32.505669+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T17:09:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: audit 2023-07-29T17:09:32.506286+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:09:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: cluster 2023-07-29T17:09:32.525141+0000 mon.a (mon.0) 682 : cluster [DBG] mgrmap e19: x(active, since 2s), standbys: y 2023-07-29T17:09:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:33 smithi129 bash[23526]: audit 2023-07-29T17:09:32.532394+0000 mon.b (mon.1) 50 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:09:33.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: cluster 2023-07-29T17:09:32.201137+0000 mgr.x (mgr.24745) 11 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:33.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: audit 2023-07-29T17:09:32.365662+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: cluster 2023-07-29T17:09:32.366022+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: cluster 2023-07-29T17:09:32.498096+0000 mon.a (mon.0) 681 : cluster [DBG] Standby manager daemon y started 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: audit 2023-07-29T17:09:32.498858+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: audit 2023-07-29T17:09:32.503782+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: audit 2023-07-29T17:09:32.505669+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: audit 2023-07-29T17:09:32.506286+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:09:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: cluster 2023-07-29T17:09:32.525141+0000 mon.a (mon.0) 682 : cluster [DBG] mgrmap e19: x(active, since 2s), standbys: y 2023-07-29T17:09:33.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[23704]: audit 2023-07-29T17:09:32.532394+0000 mon.b (mon.1) 50 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:09:33.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: cluster 2023-07-29T17:09:32.201137+0000 mgr.x (mgr.24745) 11 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:33.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: audit 2023-07-29T17:09:32.365662+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T17:09:33.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: cluster 2023-07-29T17:09:32.366022+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-29T17:09:33.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: cluster 2023-07-29T17:09:32.498096+0000 mon.a (mon.0) 681 : cluster [DBG] Standby manager daemon y started 2023-07-29T17:09:33.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: audit 2023-07-29T17:09:32.498858+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T17:09:33.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: audit 2023-07-29T17:09:32.503782+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:09:33.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: audit 2023-07-29T17:09:32.505669+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T17:09:33.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: audit 2023-07-29T17:09:32.506286+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.78:0/2864905711' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:09:33.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: cluster 2023-07-29T17:09:32.525141+0000 mon.a (mon.0) 682 : cluster [DBG] mgrmap e19: x(active, since 2s), standbys: y 2023-07-29T17:09:33.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:33 smithi078 bash[28259]: audit 2023-07-29T17:09:32.532394+0000 mon.b (mon.1) 50 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:09:34.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:34 smithi078 bash[23704]: audit 2023-07-29T17:09:33.648010+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:34.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:34 smithi078 bash[28259]: audit 2023-07-29T17:09:33.648010+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:35.167 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:34 smithi129 bash[23526]: audit 2023-07-29T17:09:33.648010+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:35.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:35 smithi078 bash[28259]: cluster 2023-07-29T17:09:34.201813+0000 mgr.x (mgr.24745) 12 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:35.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:35 smithi078 bash[28259]: cluster 2023-07-29T17:09:34.660999+0000 mon.a (mon.0) 684 : cluster [DBG] mgrmap e20: x(active, since 4s), standbys: y 2023-07-29T17:09:35.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:35 smithi078 bash[23704]: cluster 2023-07-29T17:09:34.201813+0000 mgr.x (mgr.24745) 12 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:35.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:35 smithi078 bash[23704]: cluster 2023-07-29T17:09:34.660999+0000 mon.a (mon.0) 684 : cluster [DBG] mgrmap e20: x(active, since 4s), standbys: y 2023-07-29T17:09:36.167 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:35 smithi129 bash[23526]: cluster 2023-07-29T17:09:34.201813+0000 mgr.x (mgr.24745) 12 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:36.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:35 smithi129 bash[23526]: cluster 2023-07-29T17:09:34.660999+0000 mon.a (mon.0) 684 : cluster [DBG] mgrmap e20: x(active, since 4s), standbys: y 2023-07-29T17:09:37.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:37 smithi078 bash[23704]: cluster 2023-07-29T17:09:36.202570+0000 mgr.x (mgr.24745) 13 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:37.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:37 smithi078 bash[23704]: audit 2023-07-29T17:09:37.183612+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:37.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:37 smithi078 bash[28259]: cluster 2023-07-29T17:09:36.202570+0000 mgr.x (mgr.24745) 13 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:37.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:37 smithi078 bash[28259]: audit 2023-07-29T17:09:37.183612+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:38.167 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:37 smithi129 bash[23526]: cluster 2023-07-29T17:09:36.202570+0000 mgr.x (mgr.24745) 13 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:09:38.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:37 smithi129 bash[23526]: audit 2023-07-29T17:09:37.183612+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:39.917 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:39 smithi129 bash[23526]: cluster 2023-07-29T17:09:38.203932+0000 mgr.x (mgr.24745) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 32 op/s 2023-07-29T17:09:39.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:39 smithi129 bash[23526]: audit 2023-07-29T17:09:39.584033+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:39.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:39 smithi129 bash[23526]: audit 2023-07-29T17:09:39.595186+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:39.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:39 smithi078 bash[23704]: cluster 2023-07-29T17:09:38.203932+0000 mgr.x (mgr.24745) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 32 op/s 2023-07-29T17:09:39.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:39 smithi078 bash[23704]: audit 2023-07-29T17:09:39.584033+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:39.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:39 smithi078 bash[23704]: audit 2023-07-29T17:09:39.595186+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:39.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:39 smithi078 bash[28259]: cluster 2023-07-29T17:09:38.203932+0000 mgr.x (mgr.24745) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 32 op/s 2023-07-29T17:09:39.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:39 smithi078 bash[28259]: audit 2023-07-29T17:09:39.584033+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:39.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:39 smithi078 bash[28259]: audit 2023-07-29T17:09:39.595186+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:41 smithi129 bash[23526]: audit 2023-07-29T17:09:39.974802+0000 mon.a (mon.0) 688 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:41 smithi129 bash[23526]: audit 2023-07-29T17:09:39.976208+0000 mon.b (mon.1) 51 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:41 smithi129 bash[23526]: audit 2023-07-29T17:09:39.976775+0000 mon.b (mon.1) 52 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:41 smithi129 bash[23526]: audit 2023-07-29T17:09:39.977270+0000 mon.b (mon.1) 53 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:41 smithi129 bash[23526]: audit 2023-07-29T17:09:40.876053+0000 mon.a (mon.0) 689 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:41 smithi129 bash[23526]: audit 2023-07-29T17:09:40.884238+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:41 smithi078 bash[23704]: audit 2023-07-29T17:09:39.974802+0000 mon.a (mon.0) 688 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:41 smithi078 bash[23704]: audit 2023-07-29T17:09:39.976208+0000 mon.b (mon.1) 51 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:41 smithi078 bash[23704]: audit 2023-07-29T17:09:39.976775+0000 mon.b (mon.1) 52 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:41 smithi078 bash[23704]: audit 2023-07-29T17:09:39.977270+0000 mon.b (mon.1) 53 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:41 smithi078 bash[23704]: audit 2023-07-29T17:09:40.876053+0000 mon.a (mon.0) 689 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:41 smithi078 bash[23704]: audit 2023-07-29T17:09:40.884238+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:39.974802+0000 mon.a (mon.0) 688 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:39.976208+0000 mon.b (mon.1) 51 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:39.976775+0000 mon.b (mon.1) 52 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:39.977270+0000 mon.b (mon.1) 53 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:40.876053+0000 mon.a (mon.0) 689 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:40.884238+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: cluster 2023-07-29T17:09:40.204701+0000 mgr.x (mgr.24745) 15 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 185 KiB/s wr, 25 op/s 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:40.881395+0000 mon.b (mon.1) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.782034+0000 mon.b (mon.1) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.787016+0000 mon.b (mon.1) 56 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.787508+0000 mon.b (mon.1) 57 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.790911+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.790937+0000 mon.b (mon.1) 58 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.794985+0000 mon.c (mon.2) 25 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.796561+0000 mon.b (mon.1) 59 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.796799+0000 mon.b (mon.1) 60 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:41.797027+0000 mon.b (mon.1) 61 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:42.662272+0000 mon.c (mon.2) 26 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:42.674411+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[28259]: audit 2023-07-29T17:09:42.681479+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: cluster 2023-07-29T17:09:40.204701+0000 mgr.x (mgr.24745) 15 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 185 KiB/s wr, 25 op/s 2023-07-29T17:09:42.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:40.881395+0000 mon.b (mon.1) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.782034+0000 mon.b (mon.1) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.787016+0000 mon.b (mon.1) 56 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.787508+0000 mon.b (mon.1) 57 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.790911+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.790937+0000 mon.b (mon.1) 58 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.794985+0000 mon.c (mon.2) 25 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.796561+0000 mon.b (mon.1) 59 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:42.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.796799+0000 mon.b (mon.1) 60 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:42.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:41.797027+0000 mon.b (mon.1) 61 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:42.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:42.662272+0000 mon.c (mon.2) 26 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:42.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:42.674411+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:42.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:42 smithi078 bash[23704]: audit 2023-07-29T17:09:42.681479+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:43.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: cluster 2023-07-29T17:09:40.204701+0000 mgr.x (mgr.24745) 15 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 185 KiB/s wr, 25 op/s 2023-07-29T17:09:43.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:40.881395+0000 mon.b (mon.1) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.782034+0000 mon.b (mon.1) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.787016+0000 mon.b (mon.1) 56 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.787508+0000 mon.b (mon.1) 57 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.790911+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:43.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.790937+0000 mon.b (mon.1) 58 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:43.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.794985+0000 mon.c (mon.2) 25 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T17:09:43.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.796561+0000 mon.b (mon.1) 59 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:09:43.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.796799+0000 mon.b (mon.1) 60 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:09:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:41.797027+0000 mon.b (mon.1) 61 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:09:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:42.662272+0000 mon.c (mon.2) 26 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T17:09:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:42.674411+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:42 smithi129 bash[23526]: audit 2023-07-29T17:09:42.681479+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:44.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:43 smithi129 bash[23526]: cluster 2023-07-29T17:09:42.206414+0000 mgr.x (mgr.24745) 16 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 215 KiB/s wr, 26 op/s 2023-07-29T17:09:44.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:43 smithi129 bash[23526]: audit 2023-07-29T17:09:43.475690+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:44.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:43 smithi129 bash[23526]: audit 2023-07-29T17:09:43.488930+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:44.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:43 smithi078 bash[23704]: cluster 2023-07-29T17:09:42.206414+0000 mgr.x (mgr.24745) 16 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 215 KiB/s wr, 26 op/s 2023-07-29T17:09:44.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:43 smithi078 bash[23704]: audit 2023-07-29T17:09:43.475690+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:44.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:43 smithi078 bash[23704]: audit 2023-07-29T17:09:43.488930+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:44.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:43 smithi078 bash[28259]: cluster 2023-07-29T17:09:42.206414+0000 mgr.x (mgr.24745) 16 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 215 KiB/s wr, 26 op/s 2023-07-29T17:09:44.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:43 smithi078 bash[28259]: audit 2023-07-29T17:09:43.475690+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:44.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:43 smithi078 bash[28259]: audit 2023-07-29T17:09:43.488930+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:46.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:45 smithi129 bash[23526]: cluster 2023-07-29T17:09:44.207320+0000 mgr.x (mgr.24745) 17 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 182 KiB/s wr, 22 op/s 2023-07-29T17:09:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:45 smithi078 bash[23704]: cluster 2023-07-29T17:09:44.207320+0000 mgr.x (mgr.24745) 17 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 182 KiB/s wr, 22 op/s 2023-07-29T17:09:46.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:45 smithi078 bash[28259]: cluster 2023-07-29T17:09:44.207320+0000 mgr.x (mgr.24745) 17 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 182 KiB/s wr, 22 op/s 2023-07-29T17:09:47.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:47 smithi078 bash[23704]: cluster 2023-07-29T17:09:46.207898+0000 mgr.x (mgr.24745) 18 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 179 KiB/s wr, 21 op/s 2023-07-29T17:09:47.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:47 smithi078 bash[28259]: cluster 2023-07-29T17:09:46.207898+0000 mgr.x (mgr.24745) 18 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 179 KiB/s wr, 21 op/s 2023-07-29T17:09:48.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:47 smithi129 bash[23526]: cluster 2023-07-29T17:09:46.207898+0000 mgr.x (mgr.24745) 18 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 179 KiB/s wr, 21 op/s 2023-07-29T17:09:50.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:49 smithi129 bash[23526]: cluster 2023-07-29T17:09:48.209527+0000 mgr.x (mgr.24745) 19 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 201 KiB/s wr, 24 op/s 2023-07-29T17:09:50.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:49 smithi078 bash[28259]: cluster 2023-07-29T17:09:48.209527+0000 mgr.x (mgr.24745) 19 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 201 KiB/s wr, 24 op/s 2023-07-29T17:09:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:49 smithi078 bash[23704]: cluster 2023-07-29T17:09:48.209527+0000 mgr.x (mgr.24745) 19 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 201 KiB/s wr, 24 op/s 2023-07-29T17:09:51.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:51 smithi129 bash[23526]: cluster 2023-07-29T17:09:50.210321+0000 mgr.x (mgr.24745) 20 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-29T17:09:52.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:51 smithi078 bash[28259]: cluster 2023-07-29T17:09:50.210321+0000 mgr.x (mgr.24745) 20 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-29T17:09:52.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:51 smithi078 bash[23704]: cluster 2023-07-29T17:09:50.210321+0000 mgr.x (mgr.24745) 20 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-29T17:09:53.832 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:09:53.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:53 smithi129 bash[23526]: cluster 2023-07-29T17:09:52.212035+0000 mgr.x (mgr.24745) 21 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-29T17:09:53.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:53 smithi129 bash[23526]: audit 2023-07-29T17:09:52.550920+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:53.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:53 smithi129 bash[23526]: audit 2023-07-29T17:09:52.564095+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:53.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:53 smithi078 bash[23704]: cluster 2023-07-29T17:09:52.212035+0000 mgr.x (mgr.24745) 21 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-29T17:09:53.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:53 smithi078 bash[23704]: audit 2023-07-29T17:09:52.550920+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:53.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:53 smithi078 bash[23704]: audit 2023-07-29T17:09:52.564095+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:53.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:53 smithi078 bash[28259]: cluster 2023-07-29T17:09:52.212035+0000 mgr.x (mgr.24745) 21 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-29T17:09:53.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:53 smithi078 bash[28259]: audit 2023-07-29T17:09:52.550920+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:53.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:53 smithi078 bash[28259]: audit 2023-07-29T17:09:52.564095+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.414 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:54 smithi078 systemd[1]: Stopping Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:09:54.580 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:09:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5m) 14s ago 6m 10.2M - 0.20.0 0881eb8f169f 6b73f9bccabd 2023-07-29T17:09:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (5m) 14s ago 5m 26.5M - 6.7.4 557c83e11646 a2bc911e186a 2023-07-29T17:09:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (3m) 14s ago 3m 43.9M - 3.4 24ecd6d5f14c 6685d4a4e408 2023-07-29T17:09:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443 running (33s) 14s ago 10m 460M - 18.1.3-11-g0484097b b662b4d80c86 1ad497f13c14 2023-07-29T17:09:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283 running (12m) 14s ago 12m 354M - 16.2.0 24ecd6d5f14c 97596eccd9c9 2023-07-29T17:09:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (12m) 14s ago 12m 57.9M 2048M 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (10m) 14s ago 10m 51.0M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 14s ago 10m 47.7M 2048M 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6m) 14s ago 6m 10.7M - 0.18.1 e5a616e4b9cf a6dca72e6f42 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (6m) 14s ago 6m 11.4M - 0.18.1 e5a616e4b9cf 4280de6ba851 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (9m) 14s ago 10m 53.5M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (9m) 14s ago 9m 51.6M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (9m) 14s ago 9m 46.5M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:09:54.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (8m) 14s ago 8m 52.3M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (8m) 14s ago 8m 52.0M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (7m) 14s ago 7m 49.4M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (7m) 14s ago 7m 48.9M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (7m) 14s ago 7m 52.4M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 running (5m) 14s ago 6m 25.2M - 2.18.1 de242295e225 0cfe64e114f3 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (4m) 14s ago 4m 52.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:09:54.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (4m) 14s ago 4m 52.5M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:09:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (3m) 14s ago 3m 52.3M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:09:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (3m) 14s ago 3m 52.7M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:09:54.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.646114+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.662584+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.677745+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.688173+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.704946+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.716857+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.729806+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: cephadm 2023-07-29T17:09:53.736849+0000 mgr.x (mgr.24745) 22 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: cephadm 2023-07-29T17:09:53.737780+0000 mgr.x (mgr.24745) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.745755+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: audit 2023-07-29T17:09:53.803629+0000 mgr.x (mgr.24745) 24 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: cephadm 2023-07-29T17:09:53.860450+0000 mgr.x (mgr.24745) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-07-29T17:09:54.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[23704]: cephadm 2023-07-29T17:09:53.861126+0000 mgr.x (mgr.24745) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi078 2023-07-29T17:09:54.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.646114+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.662584+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.677745+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.688173+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.704946+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.716857+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.729806+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: cephadm 2023-07-29T17:09:53.736849+0000 mgr.x (mgr.24745) 22 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: cephadm 2023-07-29T17:09:53.737780+0000 mgr.x (mgr.24745) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.745755+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: audit 2023-07-29T17:09:53.803629+0000 mgr.x (mgr.24745) 24 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: cephadm 2023-07-29T17:09:53.860450+0000 mgr.x (mgr.24745) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-07-29T17:09:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:54 smithi078 bash[28259]: cephadm 2023-07-29T17:09:53.861126+0000 mgr.x (mgr.24745) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi078 2023-07-29T17:09:54.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.646114+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.662584+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.677745+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.688173+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.704946+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.716857+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.729806+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: cephadm 2023-07-29T17:09:53.736849+0000 mgr.x (mgr.24745) 22 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: cephadm 2023-07-29T17:09:53.737780+0000 mgr.x (mgr.24745) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-07-29T17:09:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.745755+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:54.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: audit 2023-07-29T17:09:53.803629+0000 mgr.x (mgr.24745) 24 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:54.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: cephadm 2023-07-29T17:09:53.860450+0000 mgr.x (mgr.24745) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-07-29T17:09:54.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:54 smithi129 bash[23526]: cephadm 2023-07-29T17:09:53.861126+0000 mgr.x (mgr.24745) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi078 2023-07-29T17:09:55.019 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:09:55.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:09:55.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:09:55.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:55.020 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:09:55.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:09:55.022 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:09:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:09:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:09:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:09:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:09:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2023-07-29T17:09:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "1/23 daemons upgraded", 2023-07-29T17:09:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "", 2023-07-29T17:09:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:09:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:09:55.436 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:54 smithi078 docker[54586]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-node-exporter.a 2023-07-29T17:09:55.436 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:55 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:09:55.436 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:55 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-29T17:09:55.436 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:55 smithi078 systemd[1]: Stopped Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:09:55.436 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:55 smithi078 systemd[1]: Started Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:09:55.785 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:09:56.161 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[28259]: audit 2023-07-29T17:09:54.173146+0000 mgr.x (mgr.24745) 27 : audit [DBG] from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:56.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[28259]: cluster 2023-07-29T17:09:54.212777+0000 mgr.x (mgr.24745) 28 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2023-07-29T17:09:56.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[28259]: audit 2023-07-29T17:09:54.543958+0000 mgr.x (mgr.24745) 29 : audit [DBG] from='client.24869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:56.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[28259]: audit 2023-07-29T17:09:55.004312+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? 172.21.15.78:0/2837626291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:09:56.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[28259]: audit 2023-07-29T17:09:55.048811+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:56.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[28259]: audit 2023-07-29T17:09:55.060554+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:56.162 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[23704]: audit 2023-07-29T17:09:54.173146+0000 mgr.x (mgr.24745) 27 : audit [DBG] from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:56.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[23704]: cluster 2023-07-29T17:09:54.212777+0000 mgr.x (mgr.24745) 28 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2023-07-29T17:09:56.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[23704]: audit 2023-07-29T17:09:54.543958+0000 mgr.x (mgr.24745) 29 : audit [DBG] from='client.24869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:56.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[23704]: audit 2023-07-29T17:09:55.004312+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? 172.21.15.78:0/2837626291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:09:56.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[23704]: audit 2023-07-29T17:09:55.048811+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:56.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:55 smithi078 bash[23704]: audit 2023-07-29T17:09:55.060554+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:56.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:55 smithi129 bash[23526]: audit 2023-07-29T17:09:54.173146+0000 mgr.x (mgr.24745) 27 : audit [DBG] from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:56.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:55 smithi129 bash[23526]: cluster 2023-07-29T17:09:54.212777+0000 mgr.x (mgr.24745) 28 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2023-07-29T17:09:56.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:55 smithi129 bash[23526]: audit 2023-07-29T17:09:54.543958+0000 mgr.x (mgr.24745) 29 : audit [DBG] from='client.24869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:56.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:55 smithi129 bash[23526]: audit 2023-07-29T17:09:55.004312+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? 172.21.15.78:0/2837626291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:09:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:55 smithi129 bash[23526]: audit 2023-07-29T17:09:55.048811+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:55 smithi129 bash[23526]: audit 2023-07-29T17:09:55.060554+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:09:56.436 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-29T17:09:56.437 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-29T17:09:56.437 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-29T17:09:56.437 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-29T17:09:56.437 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-29T17:09:56.437 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-29T17:09:56.437 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-29T17:09:56.438 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-29T17:09:56.439 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-29T17:09:56.440 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-29T17:09:56.441 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-29T17:09:56.441 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-29T17:09:56.441 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-29T17:09:56.441 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-29T17:09:56.441 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[54684]: time="2023-07-29T17:09:56Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-29T17:09:57.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:56 smithi129 bash[23526]: cephadm 2023-07-29T17:09:55.078863+0000 mgr.x (mgr.24745) 30 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:09:57.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:56 smithi129 bash[23526]: cephadm 2023-07-29T17:09:55.100231+0000 mgr.x (mgr.24745) 31 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:09:57.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:56 smithi129 bash[23526]: audit 2023-07-29T17:09:55.366612+0000 mgr.x (mgr.24745) 32 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:56 smithi129 bash[23526]: audit 2023-07-29T17:09:55.771241+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? 172.21.15.78:0/832259345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:09:57.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[28259]: cephadm 2023-07-29T17:09:55.078863+0000 mgr.x (mgr.24745) 30 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:09:57.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[28259]: cephadm 2023-07-29T17:09:55.100231+0000 mgr.x (mgr.24745) 31 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:09:57.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[28259]: audit 2023-07-29T17:09:55.366612+0000 mgr.x (mgr.24745) 32 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:57.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[28259]: audit 2023-07-29T17:09:55.771241+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? 172.21.15.78:0/832259345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:09:57.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[23704]: cephadm 2023-07-29T17:09:55.078863+0000 mgr.x (mgr.24745) 30 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:09:57.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[23704]: cephadm 2023-07-29T17:09:55.100231+0000 mgr.x (mgr.24745) 31 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:09:57.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[23704]: audit 2023-07-29T17:09:55.366612+0000 mgr.x (mgr.24745) 32 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:09:57.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:56 smithi078 bash[23704]: audit 2023-07-29T17:09:55.771241+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? 172.21.15.78:0/832259345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:09:58.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:57 smithi129 bash[23526]: cluster 2023-07-29T17:09:56.213584+0000 mgr.x (mgr.24745) 33 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2023-07-29T17:09:58.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:57 smithi078 bash[23704]: cluster 2023-07-29T17:09:56.213584+0000 mgr.x (mgr.24745) 33 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2023-07-29T17:09:58.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:57 smithi078 bash[28259]: cluster 2023-07-29T17:09:56.213584+0000 mgr.x (mgr.24745) 33 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2023-07-29T17:10:00.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:09:59 smithi129 bash[23526]: cluster 2023-07-29T17:09:58.215312+0000 mgr.x (mgr.24745) 34 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 22 KiB/s wr, 3 op/s 2023-07-29T17:10:00.177 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:09:59 smithi078 bash[28259]: cluster 2023-07-29T17:09:58.215312+0000 mgr.x (mgr.24745) 34 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 22 KiB/s wr, 3 op/s 2023-07-29T17:10:00.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:09:59 smithi078 bash[23704]: cluster 2023-07-29T17:09:58.215312+0000 mgr.x (mgr.24745) 34 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 22 KiB/s wr, 3 op/s 2023-07-29T17:10:01.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:00 smithi078 bash[28259]: cluster 2023-07-29T17:10:00.000118+0000 mon.a (mon.0) 709 : cluster [INF] overall HEALTH_OK 2023-07-29T17:10:01.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:00 smithi078 bash[23704]: cluster 2023-07-29T17:10:00.000118+0000 mon.a (mon.0) 709 : cluster [INF] overall HEALTH_OK 2023-07-29T17:10:01.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:00 smithi129 bash[23526]: cluster 2023-07-29T17:10:00.000118+0000 mon.a (mon.0) 709 : cluster [INF] overall HEALTH_OK 2023-07-29T17:10:02.078 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:01 smithi078 bash[23704]: cluster 2023-07-29T17:10:00.216067+0000 mgr.x (mgr.24745) 35 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:02.078 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:01 smithi078 bash[28259]: cluster 2023-07-29T17:10:00.216067+0000 mgr.x (mgr.24745) 35 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:02.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:01 smithi129 bash[23526]: cluster 2023-07-29T17:10:00.216067+0000 mgr.x (mgr.24745) 35 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:03.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:03 smithi078 bash[23704]: cluster 2023-07-29T17:10:02.217727+0000 mgr.x (mgr.24745) 36 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:03.930 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:03 smithi078 bash[28259]: cluster 2023-07-29T17:10:02.217727+0000 mgr.x (mgr.24745) 36 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:03.931 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:03 smithi078 systemd[1]: Stopping Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:10:03.931 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:03 smithi078 bash[44801]: level=info ts=2023-07-29T17:10:03.734Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:10:04.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:03 smithi129 bash[23526]: cluster 2023-07-29T17:10:02.217727+0000 mgr.x (mgr.24745) 36 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:04.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:04 smithi078 docker[55069]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-alertmanager.a 2023-07-29T17:10:04.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:04 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-29T17:10:04.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:04 smithi078 systemd[1]: Stopped Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:04.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:04 smithi078 systemd[1]: Started Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:05.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[23704]: cluster 2023-07-29T17:10:04.218478+0000 mgr.x (mgr.24745) 37 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:05.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[23704]: audit 2023-07-29T17:10:04.564307+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:05.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[23704]: audit 2023-07-29T17:10:04.575099+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:05.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[23704]: cephadm 2023-07-29T17:10:04.591366+0000 mgr.x (mgr.24745) 38 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:10:05.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[23704]: cephadm 2023-07-29T17:10:04.832178+0000 mgr.x (mgr.24745) 39 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:10:05.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[28259]: cluster 2023-07-29T17:10:04.218478+0000 mgr.x (mgr.24745) 37 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:05.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[28259]: audit 2023-07-29T17:10:04.564307+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:05.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[28259]: audit 2023-07-29T17:10:04.575099+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:05.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[28259]: cephadm 2023-07-29T17:10:04.591366+0000 mgr.x (mgr.24745) 38 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:10:05.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[28259]: cephadm 2023-07-29T17:10:04.832178+0000 mgr.x (mgr.24745) 39 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:10:05.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:05 smithi129 bash[23526]: cluster 2023-07-29T17:10:04.218478+0000 mgr.x (mgr.24745) 37 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:05.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:05 smithi129 bash[23526]: audit 2023-07-29T17:10:04.564307+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:05 smithi129 bash[23526]: audit 2023-07-29T17:10:04.575099+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:05 smithi129 bash[23526]: cephadm 2023-07-29T17:10:04.591366+0000 mgr.x (mgr.24745) 38 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:10:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:05 smithi129 bash[23526]: cephadm 2023-07-29T17:10:04.832178+0000 mgr.x (mgr.24745) 39 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:10:06.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.835Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2023-07-29T17:10:06.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.835Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2023-07-29T17:10:06.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.837Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.78 port=9094 2023-07-29T17:10:06.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.839Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-29T17:10:06.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.886Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:10:06.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.886Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:10:06.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:05 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:05.891Z caller=main.go:497 msg=Listening address=:9093 2023-07-29T17:10:07.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:07 smithi078 bash[23704]: cluster 2023-07-29T17:10:06.219293+0000 mgr.x (mgr.24745) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:07.855 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:07 smithi078 bash[28259]: cluster 2023-07-29T17:10:06.219293+0000 mgr.x (mgr.24745) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:07.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:07 smithi129 bash[23526]: cluster 2023-07-29T17:10:06.219293+0000 mgr.x (mgr.24745) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:08.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:07 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:07.839Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000131912s 2023-07-29T17:10:09.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:09 smithi129 bash[23526]: cluster 2023-07-29T17:10:08.220945+0000 mgr.x (mgr.24745) 41 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:09.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:09 smithi078 bash[28259]: cluster 2023-07-29T17:10:08.220945+0000 mgr.x (mgr.24745) 41 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:09.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:09 smithi078 bash[23704]: cluster 2023-07-29T17:10:08.220945+0000 mgr.x (mgr.24745) 41 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:11.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:11 smithi129 bash[23526]: cluster 2023-07-29T17:10:10.221552+0000 mgr.x (mgr.24745) 42 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:11.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:11 smithi078 bash[23704]: cluster 2023-07-29T17:10:10.221552+0000 mgr.x (mgr.24745) 42 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:11.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:11 smithi078 bash[28259]: cluster 2023-07-29T17:10:10.221552+0000 mgr.x (mgr.24745) 42 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:13.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:13 smithi129 bash[23526]: cluster 2023-07-29T17:10:12.223292+0000 mgr.x (mgr.24745) 43 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:13.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:13 smithi078 bash[28259]: cluster 2023-07-29T17:10:12.223292+0000 mgr.x (mgr.24745) 43 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:13.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:13 smithi078 bash[23704]: cluster 2023-07-29T17:10:12.223292+0000 mgr.x (mgr.24745) 43 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:15.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:15 smithi129 bash[23526]: cluster 2023-07-29T17:10:14.224066+0000 mgr.x (mgr.24745) 44 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:15.855 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:15 smithi078 bash[23704]: cluster 2023-07-29T17:10:14.224066+0000 mgr.x (mgr.24745) 44 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:15.856 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:15 smithi078 bash[28259]: cluster 2023-07-29T17:10:14.224066+0000 mgr.x (mgr.24745) 44 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:16.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:10:15 smithi078 bash[55117]: level=info ts=2023-07-29T17:10:15.840Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000997343s 2023-07-29T17:10:17.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:17 smithi129 bash[23526]: cluster 2023-07-29T17:10:16.225088+0000 mgr.x (mgr.24745) 45 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:17.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:17 smithi078 bash[28259]: cluster 2023-07-29T17:10:16.225088+0000 mgr.x (mgr.24745) 45 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:17.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:17 smithi078 bash[23704]: cluster 2023-07-29T17:10:16.225088+0000 mgr.x (mgr.24745) 45 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:18.402 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 systemd[1]: Stopping Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:10:18.402 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=warn ts=2023-07-29T17:10:18.219Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:10:18.402 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:10:18.402 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.220Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:10:18.403 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.221Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:10:18.404 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.221Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:10:18.404 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 bash[38636]: level=info ts=2023-07-29T17:10:18.221Z caller=main.go:761 msg="See you next time!" 2023-07-29T17:10:19.317 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 docker[45337]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-prometheus.a 2023-07-29T17:10:19.317 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Succeeded. 2023-07-29T17:10:19.317 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:19.318 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:18 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:19.318 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:19 smithi129 systemd[1]: Stopping Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:10:19.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:19 smithi078 bash[23704]: cluster 2023-07-29T17:10:18.227127+0000 mgr.x (mgr.24745) 46 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:19.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:19 smithi078 bash[23704]: audit 2023-07-29T17:10:19.012366+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:19.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:19 smithi078 bash[23704]: audit 2023-07-29T17:10:19.026812+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:19.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:19 smithi078 bash[28259]: cluster 2023-07-29T17:10:18.227127+0000 mgr.x (mgr.24745) 46 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:19.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:19 smithi078 bash[28259]: audit 2023-07-29T17:10:19.012366+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:19.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:19 smithi078 bash[28259]: audit 2023-07-29T17:10:19.026812+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:19.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:19 smithi129 bash[23526]: cluster 2023-07-29T17:10:18.227127+0000 mgr.x (mgr.24745) 46 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:19.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:19 smithi129 bash[23526]: audit 2023-07-29T17:10:19.012366+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:19.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:19 smithi129 bash[23526]: audit 2023-07-29T17:10:19.026812+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:20.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:20 smithi129 bash[23526]: cephadm 2023-07-29T17:10:19.033761+0000 mgr.x (mgr.24745) 47 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-07-29T17:10:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:20 smithi129 bash[23526]: cephadm 2023-07-29T17:10:19.034250+0000 mgr.x (mgr.24745) 48 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi129 2023-07-29T17:10:20.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:20 smithi129 docker[45414]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-node-exporter.b 2023-07-29T17:10:20.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:20 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:10:20.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:20 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-29T17:10:20.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:20 smithi129 systemd[1]: Stopped Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:20.920 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:20 smithi129 systemd[1]: Started Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:20 smithi078 bash[28259]: cephadm 2023-07-29T17:10:19.033761+0000 mgr.x (mgr.24745) 47 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-07-29T17:10:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:20 smithi078 bash[28259]: cephadm 2023-07-29T17:10:19.034250+0000 mgr.x (mgr.24745) 48 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi129 2023-07-29T17:10:20.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:20 smithi078 bash[23704]: cephadm 2023-07-29T17:10:19.033761+0000 mgr.x (mgr.24745) 47 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-07-29T17:10:20.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:20 smithi078 bash[23704]: cephadm 2023-07-29T17:10:19.034250+0000 mgr.x (mgr.24745) 48 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi129 2023-07-29T17:10:21.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.164Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:10:21.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.164Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:10:21.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.164Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.164Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.164Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.164Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.166Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.166Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.169Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.170Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2023-07-29T17:10:21.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.183Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.183Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.183Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=14.030611ms 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.185Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.185Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.185Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:10:21.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:10:21.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=info ts=2023-07-29T17:10:21.186Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:10:21.673 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[45388]: level=error ts=2023-07-29T17:10:21.187Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:10:22.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: cluster 2023-07-29T17:10:20.227916+0000 mgr.x (mgr.24745) 49 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: audit 2023-07-29T17:10:20.759251+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: audit 2023-07-29T17:10:20.771394+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: cephadm 2023-07-29T17:10:20.803607+0000 mgr.x (mgr.24745) 50 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-07-29T17:10:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: cephadm 2023-07-29T17:10:20.819379+0000 mgr.x (mgr.24745) 51 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-07-29T17:10:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: audit 2023-07-29T17:10:20.907700+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: audit 2023-07-29T17:10:20.920128+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:21 smithi129 bash[23526]: audit 2023-07-29T17:10:20.927856+0000 mon.b (mon.1) 62 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:10:22.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: cluster 2023-07-29T17:10:20.227916+0000 mgr.x (mgr.24745) 49 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: audit 2023-07-29T17:10:20.759251+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: audit 2023-07-29T17:10:20.771394+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: cephadm 2023-07-29T17:10:20.803607+0000 mgr.x (mgr.24745) 50 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: cephadm 2023-07-29T17:10:20.819379+0000 mgr.x (mgr.24745) 51 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: audit 2023-07-29T17:10:20.907700+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: audit 2023-07-29T17:10:20.920128+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[28259]: audit 2023-07-29T17:10:20.927856+0000 mon.b (mon.1) 62 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: cluster 2023-07-29T17:10:20.227916+0000 mgr.x (mgr.24745) 49 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: audit 2023-07-29T17:10:20.759251+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: audit 2023-07-29T17:10:20.771394+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: cephadm 2023-07-29T17:10:20.803607+0000 mgr.x (mgr.24745) 50 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: cephadm 2023-07-29T17:10:20.819379+0000 mgr.x (mgr.24745) 51 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: audit 2023-07-29T17:10:20.907700+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: audit 2023-07-29T17:10:20.920128+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:22.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:21 smithi078 bash[23704]: audit 2023-07-29T17:10:20.927856+0000 mon.b (mon.1) 62 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:10:23.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[23526]: audit 2023-07-29T17:10:20.928479+0000 mgr.x (mgr.24745) 52 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:10:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[23526]: cephadm 2023-07-29T17:10:20.932157+0000 mgr.x (mgr.24745) 53 : cephadm [INF] Reconfiguring daemon grafana.a on smithi129 2023-07-29T17:10:23.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:22 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:10:23.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:22 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-29T17:10:23.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-29T17:10:23.171 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-29T17:10:23.172 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-29T17:10:23.173 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-29T17:10:23.174 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-29T17:10:23.174 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-29T17:10:23.174 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-29T17:10:23.174 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-29T17:10:23.174 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-29T17:10:23.174 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:10:22 smithi129 bash[45461]: time="2023-07-29T17:10:22Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-29T17:10:23.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:22 smithi078 bash[28259]: audit 2023-07-29T17:10:20.928479+0000 mgr.x (mgr.24745) 52 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:10:23.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:22 smithi078 bash[28259]: cephadm 2023-07-29T17:10:20.932157+0000 mgr.x (mgr.24745) 53 : cephadm [INF] Reconfiguring daemon grafana.a on smithi129 2023-07-29T17:10:23.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:22 smithi078 bash[23704]: audit 2023-07-29T17:10:20.928479+0000 mgr.x (mgr.24745) 52 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-29T17:10:23.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:22 smithi078 bash[23704]: cephadm 2023-07-29T17:10:20.932157+0000 mgr.x (mgr.24745) 53 : cephadm [INF] Reconfiguring daemon grafana.a on smithi129 2023-07-29T17:10:24.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:23 smithi129 bash[23526]: cluster 2023-07-29T17:10:22.229562+0000 mgr.x (mgr.24745) 54 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:24.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:23 smithi078 bash[23704]: cluster 2023-07-29T17:10:22.229562+0000 mgr.x (mgr.24745) 54 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:24.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:23 smithi078 bash[28259]: cluster 2023-07-29T17:10:22.229562+0000 mgr.x (mgr.24745) 54 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:26.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:25 smithi129 bash[23526]: cluster 2023-07-29T17:10:24.230365+0000 mgr.x (mgr.24745) 55 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:26.179 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:10:26.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:25 smithi078 bash[23704]: cluster 2023-07-29T17:10:24.230365+0000 mgr.x (mgr.24745) 55 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:26.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:25 smithi078 bash[28259]: cluster 2023-07-29T17:10:24.230365+0000 mgr.x (mgr.24745) 55 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:26.922 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:10:26.922 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 starting - - - - 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (6m) 46s ago 6m 26.5M - 6.7.4 557c83e11646 a2bc911e186a 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (4m) 46s ago 4m 43.9M - 3.4 24ecd6d5f14c 6685d4a4e408 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443 running (66s) 46s ago 10m 460M - 18.1.3-11-g0484097b b662b4d80c86 1ad497f13c14 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283 running (12m) 47s ago 12m 354M - 16.2.0 24ecd6d5f14c 97596eccd9c9 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (12m) 47s ago 12m 57.9M 2048M 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (11m) 46s ago 11m 51.0M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:10:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (11m) 47s ago 11m 47.7M 2048M 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 starting - - - - 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 starting - - - - 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (10m) 47s ago 10m 53.5M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (10m) 47s ago 10m 51.6M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (9m) 47s ago 9m 46.5M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (9m) 47s ago 9m 52.3M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (8m) 46s ago 8m 52.0M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:10:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (8m) 46s ago 8m 49.4M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (7m) 46s ago 8m 48.9M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (7m) 46s ago 7m 52.4M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 starting - - - - 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (4m) 47s ago 4m 52.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (4m) 46s ago 4m 52.5M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (4m) 47s ago 4m 52.3M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:10:26.925 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (4m) 46s ago 4m 52.7M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:10:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:10:27.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:10:27.357 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:10:27.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:27.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:10:27.357 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-29T17:10:27.357 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:10:27.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:10:27.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:27.359 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:10:27.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-29T17:10:27.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:10:27.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:10:27.359 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:10:27.722 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:10:27.722 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:10:27.722 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:10:27.722 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:10:27.722 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2023-07-29T17:10:27.722 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "1/23 daemons upgraded", 2023-07-29T17:10:27.723 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "", 2023-07-29T17:10:27.723 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:10:27.723 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:10:28.119 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:27 smithi129 bash[23526]: audit 2023-07-29T17:10:26.149834+0000 mgr.x (mgr.24745) 56 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:27 smithi129 bash[23526]: cluster 2023-07-29T17:10:26.231238+0000 mgr.x (mgr.24745) 57 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:27 smithi129 bash[23526]: audit 2023-07-29T17:10:26.520885+0000 mgr.x (mgr.24745) 58 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:27 smithi129 bash[23526]: audit 2023-07-29T17:10:26.883877+0000 mgr.x (mgr.24745) 59 : audit [DBG] from='client.24905 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:27 smithi129 bash[23526]: audit 2023-07-29T17:10:27.341252+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? 172.21.15.78:0/4055529664' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:10:28.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[23704]: audit 2023-07-29T17:10:26.149834+0000 mgr.x (mgr.24745) 56 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[23704]: cluster 2023-07-29T17:10:26.231238+0000 mgr.x (mgr.24745) 57 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:28.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[23704]: audit 2023-07-29T17:10:26.520885+0000 mgr.x (mgr.24745) 58 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[23704]: audit 2023-07-29T17:10:26.883877+0000 mgr.x (mgr.24745) 59 : audit [DBG] from='client.24905 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[23704]: audit 2023-07-29T17:10:27.341252+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? 172.21.15.78:0/4055529664' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:10:28.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[28259]: audit 2023-07-29T17:10:26.149834+0000 mgr.x (mgr.24745) 56 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[28259]: cluster 2023-07-29T17:10:26.231238+0000 mgr.x (mgr.24745) 57 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:28.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[28259]: audit 2023-07-29T17:10:26.520885+0000 mgr.x (mgr.24745) 58 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[28259]: audit 2023-07-29T17:10:26.883877+0000 mgr.x (mgr.24745) 59 : audit [DBG] from='client.24905 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:28.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:27 smithi078 bash[28259]: audit 2023-07-29T17:10:27.341252+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? 172.21.15.78:0/4055529664' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:10:29.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:28 smithi129 bash[23526]: audit 2023-07-29T17:10:27.706834+0000 mgr.x (mgr.24745) 60 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:28 smithi129 bash[23526]: audit 2023-07-29T17:10:28.103708+0000 mon.c (mon.2) 29 : audit [DBG] from='client.? 172.21.15.78:0/4199704680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:10:29.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:28 smithi078 bash[23704]: audit 2023-07-29T17:10:27.706834+0000 mgr.x (mgr.24745) 60 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:29.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:28 smithi078 bash[23704]: audit 2023-07-29T17:10:28.103708+0000 mon.c (mon.2) 29 : audit [DBG] from='client.? 172.21.15.78:0/4199704680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:10:29.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:28 smithi078 bash[28259]: audit 2023-07-29T17:10:27.706834+0000 mgr.x (mgr.24745) 60 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:29.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:28 smithi078 bash[28259]: audit 2023-07-29T17:10:28.103708+0000 mon.c (mon.2) 29 : audit [DBG] from='client.? 172.21.15.78:0/4199704680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:10:30.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:29 smithi129 bash[23526]: cluster 2023-07-29T17:10:28.232996+0000 mgr.x (mgr.24745) 61 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:30.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:29 smithi078 bash[23704]: cluster 2023-07-29T17:10:28.232996+0000 mgr.x (mgr.24745) 61 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:30.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:29 smithi078 bash[28259]: cluster 2023-07-29T17:10:28.232996+0000 mgr.x (mgr.24745) 61 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:32.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:31 smithi129 bash[23526]: cluster 2023-07-29T17:10:30.233759+0000 mgr.x (mgr.24745) 62 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:32.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:31 smithi078 bash[23704]: cluster 2023-07-29T17:10:30.233759+0000 mgr.x (mgr.24745) 62 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:32.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:31 smithi078 bash[28259]: cluster 2023-07-29T17:10:30.233759+0000 mgr.x (mgr.24745) 62 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:33.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:32 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2023-07-29T17:10:33.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:32 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:33.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:32 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:34.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:33 smithi129 bash[23526]: cluster 2023-07-29T17:10:32.235571+0000 mgr.x (mgr.24745) 63 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:34.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:33 smithi078 bash[23704]: cluster 2023-07-29T17:10:32.235571+0000 mgr.x (mgr.24745) 63 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:34.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:33 smithi078 bash[28259]: cluster 2023-07-29T17:10:32.235571+0000 mgr.x (mgr.24745) 63 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:36.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:35 smithi129 bash[23526]: cluster 2023-07-29T17:10:34.236433+0000 mgr.x (mgr.24745) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:36.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:35 smithi078 bash[23704]: cluster 2023-07-29T17:10:34.236433+0000 mgr.x (mgr.24745) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:36.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:35 smithi078 bash[28259]: cluster 2023-07-29T17:10:34.236433+0000 mgr.x (mgr.24745) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:38.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:37 smithi129 bash[23526]: cluster 2023-07-29T17:10:36.237079+0000 mgr.x (mgr.24745) 65 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:38.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:37 smithi078 bash[23704]: cluster 2023-07-29T17:10:36.237079+0000 mgr.x (mgr.24745) 65 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:38.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:37 smithi078 bash[28259]: cluster 2023-07-29T17:10:36.237079+0000 mgr.x (mgr.24745) 65 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:38.582 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.387Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:10:38.582 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.387Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.387Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.387Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.387Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.387Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.389Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.389Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:10:38.583 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.393Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.394Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.406Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.406Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.406Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.406Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=12.897181ms 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.409Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.409Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:10:38.584 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.409Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.409Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.409Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.409Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:10:38.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:10:38.586 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:10:38.586 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=info ts=2023-07-29T17:10:38.410Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:10:38.586 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:38 smithi129 bash[45640]: level=error ts=2023-07-29T17:10:38.410Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:10:40.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:39 smithi129 bash[23526]: cluster 2023-07-29T17:10:38.237975+0000 mgr.x (mgr.24745) 66 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:40.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:39 smithi078 bash[23704]: cluster 2023-07-29T17:10:38.237975+0000 mgr.x (mgr.24745) 66 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:40.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:39 smithi078 bash[28259]: cluster 2023-07-29T17:10:38.237975+0000 mgr.x (mgr.24745) 66 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:41 smithi129 bash[23526]: cluster 2023-07-29T17:10:40.238520+0000 mgr.x (mgr.24745) 67 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:42.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:41 smithi078 bash[23704]: cluster 2023-07-29T17:10:40.238520+0000 mgr.x (mgr.24745) 67 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:42.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:41 smithi078 bash[28259]: cluster 2023-07-29T17:10:40.238520+0000 mgr.x (mgr.24745) 67 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:42.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:42 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:10:42.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:42 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:10:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:43 smithi129 bash[23526]: cluster 2023-07-29T17:10:42.240226+0000 mgr.x (mgr.24745) 68 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:44.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:43 smithi078 bash[23704]: cluster 2023-07-29T17:10:42.240226+0000 mgr.x (mgr.24745) 68 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:44.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:43 smithi078 bash[28259]: cluster 2023-07-29T17:10:42.240226+0000 mgr.x (mgr.24745) 68 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:45 smithi129 bash[23526]: cluster 2023-07-29T17:10:44.240971+0000 mgr.x (mgr.24745) 69 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:46.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:45 smithi078 bash[23704]: cluster 2023-07-29T17:10:44.240971+0000 mgr.x (mgr.24745) 69 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:46.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:45 smithi078 bash[28259]: cluster 2023-07-29T17:10:44.240971+0000 mgr.x (mgr.24745) 69 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:46.607 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 systemd[1]: Stopping Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:10:46.607 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2023-07-29T17:10:46.607 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped Stream Manager" 2023-07-29T17:10:46.607 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2023-07-29T17:10:46.607 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2023-07-29T17:10:46.607 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2023-07-29T17:10:46.608 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2023-07-29T17:10:46.608 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2023-07-29T17:10:46.608 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2023-07-29T17:10:46.608 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2023-07-29T17:10:46.608 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2023-07-29T17:10:46.608 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2023-07-29T17:10:46.609 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:46 smithi129 bash[37985]: t=2023-07-29T17:10:46+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2023-07-29T17:10:47.579 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:47 smithi129 docker[45926]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-grafana.a 2023-07-29T17:10:47.579 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:47 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@grafana.a.service: Succeeded. 2023-07-29T17:10:47.579 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:47 smithi129 systemd[1]: Stopped Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:47.579 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:47 smithi129 systemd[1]: Started Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:47 smithi129 bash[23526]: cluster 2023-07-29T17:10:46.241760+0000 mgr.x (mgr.24745) 70 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:47 smithi129 bash[23526]: audit 2023-07-29T17:10:47.230229+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:48.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:47 smithi129 bash[23526]: audit 2023-07-29T17:10:47.242635+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:48.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:47 smithi129 bash[23526]: audit 2023-07-29T17:10:47.252036+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:10:48.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:47 smithi129 bash[23526]: audit 2023-07-29T17:10:47.252578+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:10:48.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:47 smithi129 bash[23526]: audit 2023-07-29T17:10:47.264807+0000 mon.b (mon.1) 64 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:10:48.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[23704]: cluster 2023-07-29T17:10:46.241760+0000 mgr.x (mgr.24745) 70 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:48.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[23704]: audit 2023-07-29T17:10:47.230229+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:48.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[23704]: audit 2023-07-29T17:10:47.242635+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:48.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[23704]: audit 2023-07-29T17:10:47.252036+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:10:48.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[23704]: audit 2023-07-29T17:10:47.252578+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:10:48.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[23704]: audit 2023-07-29T17:10:47.264807+0000 mon.b (mon.1) 64 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:10:48.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[28259]: cluster 2023-07-29T17:10:46.241760+0000 mgr.x (mgr.24745) 70 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:48.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[28259]: audit 2023-07-29T17:10:47.230229+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:48.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[28259]: audit 2023-07-29T17:10:47.242635+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:48.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[28259]: audit 2023-07-29T17:10:47.252036+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:10:48.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[28259]: audit 2023-07-29T17:10:47.252578+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:10:48.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:47 smithi078 bash[28259]: audit 2023-07-29T17:10:47.264807+0000 mon.b (mon.1) 64 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:10:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:48 smithi129 bash[23526]: cephadm 2023-07-29T17:10:47.252030+0000 mgr.x (mgr.24745) 71 : cephadm [INF] Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:10:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:48 smithi129 bash[23526]: cephadm 2023-07-29T17:10:47.266606+0000 mgr.x (mgr.24745) 72 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:10:49.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:48 smithi078 bash[23704]: cephadm 2023-07-29T17:10:47.252030+0000 mgr.x (mgr.24745) 71 : cephadm [INF] Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:10:49.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:48 smithi078 bash[23704]: cephadm 2023-07-29T17:10:47.266606+0000 mgr.x (mgr.24745) 72 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:10:49.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:48 smithi078 bash[28259]: cephadm 2023-07-29T17:10:47.252030+0000 mgr.x (mgr.24745) 71 : cephadm [INF] Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:10:49.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:48 smithi078 bash[28259]: cephadm 2023-07-29T17:10:47.266606+0000 mgr.x (mgr.24745) 72 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:10:49.433 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2023-07-29T17:10:49.433 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2023-07-29T17:10:49.433 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2023-07-29T17:10:49.433 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2023-07-29T17:10:49.433 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2023-07-29T17:10:49.433 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="App mode production" logger=settings 2023-07-29T17:10:49.434 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Initializing SqlStore" logger=server 2023-07-29T17:10:49.435 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2023-07-29T17:10:49.435 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Starting DB migration" logger=migrator 2023-07-29T17:10:49.435 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2023-07-29T17:10:49.435 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2023-07-29T17:10:49.700 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2023-07-29T17:10:50.126 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[23526]: cluster 2023-07-29T17:10:48.243590+0000 mgr.x (mgr.24745) 73 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:50.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:49 smithi129 bash[45976]: t=2023-07-29T17:10:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2023-07-29T17:10:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:49 smithi078 bash[23704]: cluster 2023-07-29T17:10:48.243590+0000 mgr.x (mgr.24745) 73 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:50.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:49 smithi078 bash[28259]: cluster 2023-07-29T17:10:48.243590+0000 mgr.x (mgr.24745) 73 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:10:50.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:50 smithi129 bash[45976]: t=2023-07-29T17:10:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2023-07-29T17:10:50.773 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:50 smithi129 bash[45976]: t=2023-07-29T17:10:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2023-07-29T17:10:50.774 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:50 smithi129 bash[45976]: t=2023-07-29T17:10:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2023-07-29T17:10:50.774 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:50 smithi129 bash[45976]: t=2023-07-29T17:10:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2023-07-29T17:10:51.034 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:50 smithi129 bash[45976]: t=2023-07-29T17:10:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2023-07-29T17:10:51.035 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:50 smithi129 bash[45976]: t=2023-07-29T17:10:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2023-07-29T17:10:51.349 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:51 smithi129 bash[45976]: t=2023-07-29T17:10:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-29T17:10:51.350 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:51 smithi129 bash[45976]: t=2023-07-29T17:10:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2023-07-29T17:10:51.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:51 smithi129 bash[45976]: t=2023-07-29T17:10:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2023-07-29T17:10:52.147 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:51 smithi129 bash[23526]: cluster 2023-07-29T17:10:50.244418+0000 mgr.x (mgr.24745) 74 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:52.147 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:51 smithi129 bash[45976]: t=2023-07-29T17:10:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2023-07-29T17:10:52.147 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:51 smithi129 bash[45976]: t=2023-07-29T17:10:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2023-07-29T17:10:52.147 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:52 smithi129 bash[45976]: t=2023-07-29T17:10:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2023-07-29T17:10:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:51 smithi078 bash[23704]: cluster 2023-07-29T17:10:50.244418+0000 mgr.x (mgr.24745) 74 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:51 smithi078 bash[28259]: cluster 2023-07-29T17:10:50.244418+0000 mgr.x (mgr.24745) 74 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:10:52.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:52 smithi129 bash[45976]: t=2023-07-29T17:10:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2023-07-29T17:10:52.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:52 smithi129 bash[45976]: t=2023-07-29T17:10:52+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2023-07-29T17:10:52.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:52 smithi129 bash[45976]: t=2023-07-29T17:10:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2023-07-29T17:10:52.920 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:52 smithi129 bash[45976]: t=2023-07-29T17:10:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2023-07-29T17:10:52.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:52 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2023-07-29T17:10:52.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:52 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:52.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:52 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:10:53.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:53 smithi129 bash[45976]: t=2023-07-29T17:10:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2023-07-29T17:10:53.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:53 smithi129 bash[45976]: t=2023-07-29T17:10:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2023-07-29T17:10:53.881 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:53 smithi129 bash[45976]: t=2023-07-29T17:10:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2023-07-29T17:10:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:53 smithi129 bash[23526]: cluster 2023-07-29T17:10:52.246121+0000 mgr.x (mgr.24745) 75 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:10:54.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[45976]: t=2023-07-29T17:10:54+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2023-07-29T17:10:54.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:53 smithi078 bash[23704]: cluster 2023-07-29T17:10:52.246121+0000 mgr.x (mgr.24745) 75 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:10:54.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:53 smithi078 bash[28259]: cluster 2023-07-29T17:10:52.246121+0000 mgr.x (mgr.24745) 75 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:10:54.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[45976]: t=2023-07-29T17:10:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2023-07-29T17:10:55.238 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[45976]: t=2023-07-29T17:10:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2023-07-29T17:10:55.239 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:53.984970+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.239 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:53.996626+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.239 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.052523+0000 mon.b (mon.1) 65 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:10:55.239 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.053350+0000 mgr.x (mgr.24745) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:10:55.240 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.055390+0000 mon.b (mon.1) 66 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2023-07-29T17:10:55.240 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.055993+0000 mgr.x (mgr.24745) 77 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2023-07-29T17:10:55.240 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.063713+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.240 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.078478+0000 mon.b (mon.1) 67 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:10:55.240 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.079518+0000 mgr.x (mgr.24745) 78 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:10:55.240 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.088198+0000 mon.b (mon.1) 68 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.089037+0000 mgr.x (mgr.24745) 79 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.098428+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.114362+0000 mon.b (mon.1) 69 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.115069+0000 mgr.x (mgr.24745) 80 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.120481+0000 mon.b (mon.1) 70 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.121104+0000 mgr.x (mgr.24745) 81 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2023-07-29T17:10:55.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.130759+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.242 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.145805+0000 mon.b (mon.1) 71 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:10:55.242 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.146912+0000 mgr.x (mgr.24745) 82 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:10:55.242 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.166522+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:10:55.242 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.167264+0000 mgr.x (mgr.24745) 83 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:10:55.242 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.179842+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:10:55.242 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: audit 2023-07-29T17:10:54.189500+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.243 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: cephadm 2023-07-29T17:10:54.194148+0000 mgr.x (mgr.24745) 84 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:10:55.243 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: cephadm 2023-07-29T17:10:54.195301+0000 mgr.x (mgr.24745) 85 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:10:55.243 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:54 smithi129 bash[23526]: cluster 2023-07-29T17:10:54.246797+0000 mgr.x (mgr.24745) 86 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:55.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:53.984970+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:53.996626+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.052523+0000 mon.b (mon.1) 65 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:10:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.053350+0000 mgr.x (mgr.24745) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:10:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.055390+0000 mon.b (mon.1) 66 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2023-07-29T17:10:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.055993+0000 mgr.x (mgr.24745) 77 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2023-07-29T17:10:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.063713+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.078478+0000 mon.b (mon.1) 67 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.079518+0000 mgr.x (mgr.24745) 78 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.088198+0000 mon.b (mon.1) 68 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.089037+0000 mgr.x (mgr.24745) 79 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.098428+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.114362+0000 mon.b (mon.1) 69 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.115069+0000 mgr.x (mgr.24745) 80 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:10:55.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.120481+0000 mon.b (mon.1) 70 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2023-07-29T17:10:55.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.121104+0000 mgr.x (mgr.24745) 81 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2023-07-29T17:10:55.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.130759+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.145805+0000 mon.b (mon.1) 71 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:10:55.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.146912+0000 mgr.x (mgr.24745) 82 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:10:55.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:53.984970+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:53.996626+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.052523+0000 mon.b (mon.1) 65 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.053350+0000 mgr.x (mgr.24745) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.055390+0000 mon.b (mon.1) 66 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.055993+0000 mgr.x (mgr.24745) 77 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.063713+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.078478+0000 mon.b (mon.1) 67 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.079518+0000 mgr.x (mgr.24745) 78 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.088198+0000 mon.b (mon.1) 68 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.089037+0000 mgr.x (mgr.24745) 79 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.098428+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.114362+0000 mon.b (mon.1) 69 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.115069+0000 mgr.x (mgr.24745) 80 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.120481+0000 mon.b (mon.1) 70 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2023-07-29T17:10:55.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.121104+0000 mgr.x (mgr.24745) 81 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2023-07-29T17:10:55.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.130759+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.145805+0000 mon.b (mon.1) 71 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:10:55.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.146912+0000 mgr.x (mgr.24745) 82 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:10:55.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.166522+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:10:55.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.167264+0000 mgr.x (mgr.24745) 83 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.179842+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: audit 2023-07-29T17:10:54.189500+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: cephadm 2023-07-29T17:10:54.194148+0000 mgr.x (mgr.24745) 84 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: cephadm 2023-07-29T17:10:54.195301+0000 mgr.x (mgr.24745) 85 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[23704]: cluster 2023-07-29T17:10:54.246797+0000 mgr.x (mgr.24745) 86 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.166522+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:10:55.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.167264+0000 mgr.x (mgr.24745) 83 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:10:55.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.179842+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:10:55.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: audit 2023-07-29T17:10:54.189500+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:55.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: cephadm 2023-07-29T17:10:54.194148+0000 mgr.x (mgr.24745) 84 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:10:55.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: cephadm 2023-07-29T17:10:54.195301+0000 mgr.x (mgr.24745) 85 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:10:55.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:54 smithi078 bash[28259]: cluster 2023-07-29T17:10:54.246797+0000 mgr.x (mgr.24745) 86 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:55.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[45976]: t=2023-07-29T17:10:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2023-07-29T17:10:56.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[45976]: t=2023-07-29T17:10:55+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2023-07-29T17:10:56.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:56 smithi129 bash[45976]: t=2023-07-29T17:10:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2023-07-29T17:10:56.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.962Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:10:56.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.962Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:10:56.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.962Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:10:56.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.962Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:10:56.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.962Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:10:56.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.962Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:10:56.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.964Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:10:56.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.964Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:10:56.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.971Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:10:56.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.971Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.985Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.985Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.985Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.985Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.985Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=14.755152ms 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:10:56.085 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:10:56.086 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:10:56.087 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:10:56.087 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=info ts=2023-07-29T17:10:55.988Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:10:56.087 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:55 smithi129 bash[46220]: level=error ts=2023-07-29T17:10:55.988Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:10:56.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:56 smithi129 bash[45976]: t=2023-07-29T17:10:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2023-07-29T17:10:56.687 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:56 smithi129 bash[45976]: t=2023-07-29T17:10:56+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2023-07-29T17:10:56.688 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:56 smithi129 bash[45976]: t=2023-07-29T17:10:56+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2023-07-29T17:10:56.975 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:56 smithi129 bash[45976]: t=2023-07-29T17:10:56+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2023-07-29T17:10:57.642 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[45976]: t=2023-07-29T17:10:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2023-07-29T17:10:57.643 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[45976]: t=2023-07-29T17:10:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2023-07-29T17:10:57.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: cluster 2023-07-29T17:10:56.247759+0000 mgr.x (mgr.24745) 87 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:57.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: cephadm 2023-07-29T17:10:56.415335+0000 mgr.x (mgr.24745) 88 : cephadm [INF] Upgrade: Updating mgr.y 2023-07-29T17:10:57.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: audit 2023-07-29T17:10:56.423870+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:57.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: audit 2023-07-29T17:10:56.427085+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:10:57.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: audit 2023-07-29T17:10:56.427924+0000 mon.b (mon.1) 74 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:10:57.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: audit 2023-07-29T17:10:56.430518+0000 mon.b (mon.1) 75 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:10:57.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: audit 2023-07-29T17:10:56.432379+0000 mon.b (mon.1) 76 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:10:57.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[23526]: cephadm 2023-07-29T17:10:56.433996+0000 mgr.x (mgr.24745) 89 : cephadm [INF] Deploying daemon mgr.y on smithi078 2023-07-29T17:10:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: cluster 2023-07-29T17:10:56.247759+0000 mgr.x (mgr.24745) 87 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:57.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: cephadm 2023-07-29T17:10:56.415335+0000 mgr.x (mgr.24745) 88 : cephadm [INF] Upgrade: Updating mgr.y 2023-07-29T17:10:57.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: audit 2023-07-29T17:10:56.423870+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:57.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: audit 2023-07-29T17:10:56.427085+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:10:57.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: audit 2023-07-29T17:10:56.427924+0000 mon.b (mon.1) 74 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:10:57.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: audit 2023-07-29T17:10:56.430518+0000 mon.b (mon.1) 75 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:10:57.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: audit 2023-07-29T17:10:56.432379+0000 mon.b (mon.1) 76 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:10:57.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[23704]: cephadm 2023-07-29T17:10:56.433996+0000 mgr.x (mgr.24745) 89 : cephadm [INF] Deploying daemon mgr.y on smithi078 2023-07-29T17:10:57.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: cluster 2023-07-29T17:10:56.247759+0000 mgr.x (mgr.24745) 87 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:57.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: cephadm 2023-07-29T17:10:56.415335+0000 mgr.x (mgr.24745) 88 : cephadm [INF] Upgrade: Updating mgr.y 2023-07-29T17:10:57.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: audit 2023-07-29T17:10:56.423870+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:10:57.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: audit 2023-07-29T17:10:56.427085+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:10:57.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: audit 2023-07-29T17:10:56.427924+0000 mon.b (mon.1) 74 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:10:57.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: audit 2023-07-29T17:10:56.430518+0000 mon.b (mon.1) 75 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:10:57.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: audit 2023-07-29T17:10:56.432379+0000 mon.b (mon.1) 76 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:10:57.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:57 smithi078 bash[28259]: cephadm 2023-07-29T17:10:56.433996+0000 mgr.x (mgr.24745) 89 : cephadm [INF] Deploying daemon mgr.y on smithi078 2023-07-29T17:10:57.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:57 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:10:57.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:10:57 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:10:57.896 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[45976]: t=2023-07-29T17:10:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2023-07-29T17:10:57.897 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[45976]: t=2023-07-29T17:10:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2023-07-29T17:10:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[45976]: t=2023-07-29T17:10:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2023-07-29T17:10:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:57 smithi129 bash[45976]: t=2023-07-29T17:10:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-29T17:10:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2023-07-29T17:10:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2023-07-29T17:10:58.516 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:10:58.516 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2023-07-29T17:10:58.516 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2023-07-29T17:10:58.517 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2023-07-29T17:10:58.517 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2023-07-29T17:10:58.895 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2023-07-29T17:10:58.896 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2023-07-29T17:10:58.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:10:58 smithi078 systemd[1]: Stopping Ceph mgr.y for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:10:59.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2023-07-29T17:10:59.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:58 smithi129 bash[45976]: t=2023-07-29T17:10:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2023-07-29T17:10:59.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2023-07-29T17:10:59.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2023-07-29T17:10:59.259 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:10:59.260 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 starting - - - - 2023-07-29T17:10:59.260 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 starting - - - - 2023-07-29T17:10:59.260 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 starting - - - - 2023-07-29T17:10:59.260 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443 running (98s) 79s ago 11m 460M - 18.1.3-11-g0484097b b662b4d80c86 1ad497f13c14 2023-07-29T17:10:59.260 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283 running (13m) 79s ago 13m 354M - 16.2.0 24ecd6d5f14c 97596eccd9c9 2023-07-29T17:10:59.260 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (13m) 79s ago 13m 57.9M 2048M 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (11m) 79s ago 11m 51.0M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (11m) 79s ago 11m 47.7M 2048M 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 starting - - - - 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 starting - - - - 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (11m) 79s ago 11m 53.5M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (10m) 79s ago 10m 51.6M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (10m) 79s ago 10m 46.5M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:10:59.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (9m) 79s ago 9m 52.3M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (9m) 79s ago 9m 52.0M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (8m) 79s ago 9m 49.4M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (8m) 79s ago 8m 48.9M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (8m) 79s ago 8m 52.4M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 starting - - - - 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (5m) 79s ago 5m 52.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (5m) 79s ago 5m 52.5M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:10:59.262 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (4m) 79s ago 4m 52.3M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:10:59.263 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (4m) 79s ago 4m 52.7M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:10:59.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[23526]: cluster 2023-07-29T17:10:58.248775+0000 mgr.x (mgr.24745) 90 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:59.445 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2023-07-29T17:10:59.445 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2023-07-29T17:10:59.446 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2023-07-29T17:10:59.698 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[23526]: audit 2023-07-29T17:10:58.486127+0000 mgr.x (mgr.24745) 91 : audit [DBG] from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:59.698 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[23526]: audit 2023-07-29T17:10:58.488754+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? 172.21.15.129:0/2078772692' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:10:59.698 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[23526]: audit 2023-07-29T17:10:58.869226+0000 mgr.x (mgr.24745) 92 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:59.699 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2023-07-29T17:10:59.699 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2023-07-29T17:10:59.699 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2023-07-29T17:10:59.699 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2023-07-29T17:10:59.699 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2023-07-29T17:10:59.700 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:10:59.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:10:59.702 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:10:59.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-29T17:10:59.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:10:59.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:10:59.703 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:10:59.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[23704]: cluster 2023-07-29T17:10:58.248775+0000 mgr.x (mgr.24745) 90 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:59.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[23704]: audit 2023-07-29T17:10:58.486127+0000 mgr.x (mgr.24745) 91 : audit [DBG] from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:59.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[23704]: audit 2023-07-29T17:10:58.488754+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? 172.21.15.129:0/2078772692' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:10:59.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[23704]: audit 2023-07-29T17:10:58.869226+0000 mgr.x (mgr.24745) 92 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:59.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[28259]: cluster 2023-07-29T17:10:58.248775+0000 mgr.x (mgr.24745) 90 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-07-29T17:10:59.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[28259]: audit 2023-07-29T17:10:58.486127+0000 mgr.x (mgr.24745) 91 : audit [DBG] from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:10:59.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[28259]: audit 2023-07-29T17:10:58.488754+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? 172.21.15.129:0/2078772692' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:10:59.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[28259]: audit 2023-07-29T17:10:58.869226+0000 mgr.x (mgr.24745) 92 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:00.008 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2023-07-29T17:11:00.008 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2023-07-29T17:11:00.008 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2023-07-29T17:11:00.059 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:10:59 smithi078 bash[55621]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mgr.y 2023-07-29T17:11:00.059 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:10:59 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:11:00.060 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:10:59 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service: Failed with result 'exit-code'. 2023-07-29T17:11:00.060 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:10:59 smithi078 systemd[1]: Stopped Ceph mgr.y for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:00.060 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:00 smithi078 systemd[1]: Started Ceph mgr.y for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "1/23 daemons upgraded", 2023-07-29T17:11:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons", 2023-07-29T17:11:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:11:00.084 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:11:00.285 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:10:59 smithi129 bash[45976]: t=2023-07-29T17:10:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2023-07-29T17:11:00.285 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-29T17:11:00.285 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2023-07-29T17:11:00.286 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-29T17:11:00.533 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:11:00.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[23526]: audit 2023-07-29T17:10:59.222516+0000 mgr.x (mgr.24745) 93 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:00.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[23526]: audit 2023-07-29T17:10:59.684461+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? 172.21.15.78:0/2073988498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:11:00.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[23526]: audit 2023-07-29T17:11:00.084360+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:00.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[23526]: audit 2023-07-29T17:11:00.097035+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:00.648 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2023-07-29T17:11:00.648 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-29T17:11:00.648 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-29T17:11:00.649 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2023-07-29T17:11:00.649 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2023-07-29T17:11:00.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2023-07-29T17:11:00.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2023-07-29T17:11:00.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2023-07-29T17:11:00.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[23704]: audit 2023-07-29T17:10:59.222516+0000 mgr.x (mgr.24745) 93 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:00.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[23704]: audit 2023-07-29T17:10:59.684461+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? 172.21.15.78:0/2073988498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:11:00.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[23704]: audit 2023-07-29T17:11:00.084360+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:00.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[23704]: audit 2023-07-29T17:11:00.097035+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[28259]: audit 2023-07-29T17:10:59.222516+0000 mgr.x (mgr.24745) 93 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[28259]: audit 2023-07-29T17:10:59.684461+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? 172.21.15.78:0/2073988498' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:11:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[28259]: audit 2023-07-29T17:11:00.084360+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:00 smithi078 bash[28259]: audit 2023-07-29T17:11:00.097035+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:01.233 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2023-07-29T17:11:01.233 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:00 smithi129 bash[45976]: t=2023-07-29T17:11:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2023-07-29T17:11:01.234 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2023-07-29T17:11:01.234 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2023-07-29T17:11:01.234 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2023-07-29T17:11:01.502 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[23526]: audit 2023-07-29T17:11:00.067896+0000 mgr.x (mgr.24745) 94 : audit [DBG] from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:01.502 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[23526]: cluster 2023-07-29T17:11:00.249452+0000 mgr.x (mgr.24745) 95 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-29T17:11:01.502 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[23526]: audit 2023-07-29T17:11:00.517632+0000 mon.a (mon.0) 731 : audit [DBG] from='client.? 172.21.15.78:0/484808980' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:11:01.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2023-07-29T17:11:01.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2023-07-29T17:11:01.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-29T17:11:01.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[55873]: debug 2023-07-29T17:11:01.500+0000 7fb303e4d700 1 -- 172.21.15.78:0/3414347145 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644259e71e0 con 0x564426766800 2023-07-29T17:11:01.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[23704]: audit 2023-07-29T17:11:00.067896+0000 mgr.x (mgr.24745) 94 : audit [DBG] from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:01.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[23704]: cluster 2023-07-29T17:11:00.249452+0000 mgr.x (mgr.24745) 95 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-29T17:11:01.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[23704]: audit 2023-07-29T17:11:00.517632+0000 mon.a (mon.0) 731 : audit [DBG] from='client.? 172.21.15.78:0/484808980' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:11:01.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[28259]: audit 2023-07-29T17:11:00.067896+0000 mgr.x (mgr.24745) 94 : audit [DBG] from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:01.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[28259]: cluster 2023-07-29T17:11:00.249452+0000 mgr.x (mgr.24745) 95 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-29T17:11:01.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[28259]: audit 2023-07-29T17:11:00.517632+0000 mon.a (mon.0) 731 : audit [DBG] from='client.? 172.21.15.78:0/484808980' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:11:01.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2023-07-29T17:11:01.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2023-07-29T17:11:01.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2023-07-29T17:11:01.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2023-07-29T17:11:01.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2023-07-29T17:11:02.105 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2023-07-29T17:11:02.106 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:01 smithi129 bash[45976]: t=2023-07-29T17:11:01+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2023-07-29T17:11:02.106 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2023-07-29T17:11:02.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:01 smithi078 bash[55873]: debug 2023-07-29T17:11:01.668+0000 7fb30df9b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T17:11:02.393 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2023-07-29T17:11:02.393 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2023-07-29T17:11:02.393 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2023-07-29T17:11:02.393 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2023-07-29T17:11:02.644 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2023-07-29T17:11:02.644 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2023-07-29T17:11:02.645 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2023-07-29T17:11:02.645 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2023-07-29T17:11:02.904 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2023-07-29T17:11:02.905 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2023-07-29T17:11:02.905 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2023-07-29T17:11:02.905 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2023-07-29T17:11:02.905 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2023-07-29T17:11:02.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:02 smithi078 bash[55873]: debug 2023-07-29T17:11:02.552+0000 7fb30df9b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T17:11:02.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:02 smithi078 bash[55873]: debug 2023-07-29T17:11:02.652+0000 7fb30df9b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T17:11:02.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:02 smithi078 bash[55873]: debug 2023-07-29T17:11:02.748+0000 7fb30df9b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T17:11:03.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:02 smithi129 bash[45976]: t=2023-07-29T17:11:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2023-07-29T17:11:03.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2023-07-29T17:11:03.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2023-07-29T17:11:03.372 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:03 smithi078 bash[55873]: debug 2023-07-29T17:11:03.008+0000 7fb30df9b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:11:03.548 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2023-07-29T17:11:03.548 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2023-07-29T17:11:03.549 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2023-07-29T17:11:03.549 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2023-07-29T17:11:03.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:03 smithi078 bash[23704]: cluster 2023-07-29T17:11:02.251149+0000 mgr.x (mgr.24745) 96 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:03.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:03 smithi078 bash[28259]: cluster 2023-07-29T17:11:02.251149+0000 mgr.x (mgr.24745) 96 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:03.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:03 smithi078 bash[55873]: debug 2023-07-29T17:11:03.352+0000 7fb30df9b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T17:11:03.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:03 smithi078 bash[55873]: debug 2023-07-29T17:11:03.508+0000 7fb30df9b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T17:11:03.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[23526]: cluster 2023-07-29T17:11:02.251149+0000 mgr.x (mgr.24745) 96 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:03.852 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-29T17:11:03.852 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-29T17:11:03.852 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2023-07-29T17:11:03.853 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2023-07-29T17:11:03.853 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2023-07-29T17:11:04.103 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2023-07-29T17:11:04.104 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-29T17:11:04.104 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:03 smithi129 bash[45976]: t=2023-07-29T17:11:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2023-07-29T17:11:04.104 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2023-07-29T17:11:04.104 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-29T17:11:04.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:03 smithi078 bash[55873]: debug 2023-07-29T17:11:03.716+0000 7fb30df9b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T17:11:04.397 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2023-07-29T17:11:04.397 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2023-07-29T17:11:04.397 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2023-07-29T17:11:04.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2023-07-29T17:11:04.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2023-07-29T17:11:04.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2023-07-29T17:11:04.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2023-07-29T17:11:04.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2023-07-29T17:11:04.985 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2023-07-29T17:11:04.985 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2023-07-29T17:11:04.985 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2023-07-29T17:11:04.985 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2023-07-29T17:11:05.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:04 smithi078 bash[55873]: debug 2023-07-29T17:11:04.720+0000 7fb30df9b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T17:11:05.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:04 smithi078 bash[55873]: debug 2023-07-29T17:11:04.824+0000 7fb30df9b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T17:11:05.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:04 smithi078 bash[55873]: debug 2023-07-29T17:11:04.940+0000 7fb30df9b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T17:11:05.253 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:04 smithi129 bash[45976]: t=2023-07-29T17:11:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2023-07-29T17:11:05.253 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2023-07-29T17:11:05.253 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2023-07-29T17:11:05.253 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2023-07-29T17:11:05.514 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2023-07-29T17:11:05.514 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2023-07-29T17:11:05.514 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2023-07-29T17:11:05.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:05 smithi078 bash[28259]: cluster 2023-07-29T17:11:04.251984+0000 mgr.x (mgr.24745) 97 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-07-29T17:11:05.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:05 smithi078 bash[23704]: cluster 2023-07-29T17:11:04.251984+0000 mgr.x (mgr.24745) 97 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-07-29T17:11:05.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:05 smithi078 bash[55873]: debug 2023-07-29T17:11:05.412+0000 7fb30df9b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T17:11:05.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:05 smithi078 bash[55873]: debug 2023-07-29T17:11:05.644+0000 7fb30df9b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T17:11:05.783 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[23526]: cluster 2023-07-29T17:11:04.251984+0000 mgr.x (mgr.24745) 97 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-07-29T17:11:05.783 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2023-07-29T17:11:05.783 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-29T17:11:05.784 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2023-07-29T17:11:05.784 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2023-07-29T17:11:05.784 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2023-07-29T17:11:05.784 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2023-07-29T17:11:06.042 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2023-07-29T17:11:06.043 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:05 smithi129 bash[45976]: t=2023-07-29T17:11:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2023-07-29T17:11:06.303 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2023-07-29T17:11:06.303 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2023-07-29T17:11:06.303 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2023-07-29T17:11:06.303 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-29T17:11:06.437 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:06 smithi078 bash[55873]: debug 2023-07-29T17:11:06.016+0000 7fb30df9b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T17:11:06.585 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2023-07-29T17:11:06.585 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2023-07-29T17:11:06.585 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-29T17:11:06.585 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2023-07-29T17:11:06.877 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2023-07-29T17:11:06.877 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2023-07-29T17:11:06.877 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2023-07-29T17:11:06.877 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2023-07-29T17:11:06.878 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2023-07-29T17:11:06.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:06 smithi078 bash[55873]: debug 2023-07-29T17:11:06.516+0000 7fb30df9b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:11:07.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2023-07-29T17:11:07.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2023-07-29T17:11:07.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:06 smithi129 bash[45976]: t=2023-07-29T17:11:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2023-07-29T17:11:07.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2023-07-29T17:11:07.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2023-07-29T17:11:07.437 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[55873]: debug 2023-07-29T17:11:07.120+0000 7fb30df9b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T17:11:07.437 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[55873]: debug 2023-07-29T17:11:07.260+0000 7fb30df9b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T17:11:07.490 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2023-07-29T17:11:07.491 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2023-07-29T17:11:07.491 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2023-07-29T17:11:07.491 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2023-07-29T17:11:07.491 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2023-07-29T17:11:07.793 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[23526]: cluster 2023-07-29T17:11:06.252785+0000 mgr.x (mgr.24745) 98 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:07.793 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[23526]: audit 2023-07-29T17:11:07.150537+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:07.793 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[23526]: audit 2023-07-29T17:11:07.162403+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:07.794 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2023-07-29T17:11:07.794 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-29T17:11:07.794 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2023-07-29T17:11:07.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[28259]: cluster 2023-07-29T17:11:06.252785+0000 mgr.x (mgr.24745) 98 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:07.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[28259]: audit 2023-07-29T17:11:07.150537+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:07.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[28259]: audit 2023-07-29T17:11:07.162403+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:07.927 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[55873]: debug 2023-07-29T17:11:07.636+0000 7fb30df9b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T17:11:07.927 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[55873]: debug 2023-07-29T17:11:07.756+0000 7fb30df9b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T17:11:07.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[23704]: cluster 2023-07-29T17:11:06.252785+0000 mgr.x (mgr.24745) 98 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:07.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[23704]: audit 2023-07-29T17:11:07.150537+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:07.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[23704]: audit 2023-07-29T17:11:07.162403+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.105 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:07 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2023-07-29T17:11:08.105 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:07 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:08.105 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:07 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:08.105 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2023-07-29T17:11:08.105 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2023-07-29T17:11:08.106 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2023-07-29T17:11:08.106 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:07 smithi129 bash[45976]: t=2023-07-29T17:11:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2023-07-29T17:11:08.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:07 smithi078 bash[55873]: debug 2023-07-29T17:11:07.904+0000 7fb30df9b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T17:11:08.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[55873]: debug 2023-07-29T17:11:08.020+0000 7fb30df9b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T17:11:08.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[55873]: debug 2023-07-29T17:11:08.144+0000 7fb30df9b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T17:11:08.470 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[45976]: t=2023-07-29T17:11:08+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2023-07-29T17:11:08.683 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[55873]: debug 2023-07-29T17:11:08.336+0000 7fb30df9b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T17:11:08.683 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[55873]: debug 2023-07-29T17:11:08.464+0000 7fb30df9b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T17:11:08.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[45976]: t=2023-07-29T17:11:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2023-07-29T17:11:08.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[23526]: audit 2023-07-29T17:11:07.666594+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[23526]: audit 2023-07-29T17:11:07.677981+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[23526]: audit 2023-07-29T17:11:08.477177+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[23526]: audit 2023-07-29T17:11:08.492473+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[28259]: audit 2023-07-29T17:11:07.666594+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[28259]: audit 2023-07-29T17:11:07.677981+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[28259]: audit 2023-07-29T17:11:08.477177+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[28259]: audit 2023-07-29T17:11:08.492473+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[23704]: audit 2023-07-29T17:11:07.666594+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[23704]: audit 2023-07-29T17:11:07.677981+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[23704]: audit 2023-07-29T17:11:08.477177+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:08 smithi078 bash[23704]: audit 2023-07-29T17:11:08.492473+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:09.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:08 smithi129 bash[45976]: t=2023-07-29T17:11:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2023-07-29T17:11:09.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[45976]: t=2023-07-29T17:11:09+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2023-07-29T17:11:09.698 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[45976]: t=2023-07-29T17:11:09+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2023-07-29T17:11:09.698 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[45976]: t=2023-07-29T17:11:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2023-07-29T17:11:09.979 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[23526]: cluster 2023-07-29T17:11:08.253655+0000 mgr.x (mgr.24745) 99 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:09.980 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[45976]: t=2023-07-29T17:11:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-29T17:11:09.980 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[45976]: t=2023-07-29T17:11:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-29T17:11:10.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:11:09 smithi078 bash[55873]: debug 2023-07-29T17:11:09.688+0000 7fb30df9b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T17:11:10.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:09 smithi078 bash[23704]: cluster 2023-07-29T17:11:08.253655+0000 mgr.x (mgr.24745) 99 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:10.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:09 smithi078 bash[28259]: cluster 2023-07-29T17:11:08.253655+0000 mgr.x (mgr.24745) 99 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:10.256 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:09 smithi129 bash[45976]: t=2023-07-29T17:11:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-29T17:11:10.256 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[45976]: t=2023-07-29T17:11:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-29T17:11:10.257 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[45976]: t=2023-07-29T17:11:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2023-07-29T17:11:10.597 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[45976]: t=2023-07-29T17:11:10+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2023-07-29T17:11:10.598 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[45976]: t=2023-07-29T17:11:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2023-07-29T17:11:10.598 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.269Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:11:10.598 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.269Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:11:10.598 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.269Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:11:10.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.269Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:11:10.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.269Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:11:10.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.269Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:11:10.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.273Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:11:10.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.273Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:11:10.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.280Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.280Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.291Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.291Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.292Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.292Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.294Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.294Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=13.923332ms 2023-07-29T17:11:10.600 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:11:10.601 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:11:10.602 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:11:10.602 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:11:10.602 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:11:10.602 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:11:10.602 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=info ts=2023-07-29T17:11:10.297Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:11:10.602 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[47201]: level=error ts=2023-07-29T17:11:10.298Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:11:10.851 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[45976]: t=2023-07-29T17:11:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2023-07-29T17:11:10.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[23526]: cluster 2023-07-29T17:11:09.713425+0000 mon.a (mon.0) 738 : cluster [DBG] Standby manager daemon y restarted 2023-07-29T17:11:10.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[23526]: cluster 2023-07-29T17:11:09.713554+0000 mon.a (mon.0) 739 : cluster [DBG] Standby manager daemon y started 2023-07-29T17:11:10.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[23526]: audit 2023-07-29T17:11:09.716934+0000 mon.a (mon.0) 740 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T17:11:10.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[23526]: audit 2023-07-29T17:11:09.718063+0000 mon.a (mon.0) 741 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:11:10.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[23526]: audit 2023-07-29T17:11:09.719724+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T17:11:10.852 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[23526]: audit 2023-07-29T17:11:09.720354+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:11:10.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:10 smithi129 bash[45976]: t=2023-07-29T17:11:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-29T17:11:11.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[23704]: cluster 2023-07-29T17:11:09.713425+0000 mon.a (mon.0) 738 : cluster [DBG] Standby manager daemon y restarted 2023-07-29T17:11:11.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[23704]: cluster 2023-07-29T17:11:09.713554+0000 mon.a (mon.0) 739 : cluster [DBG] Standby manager daemon y started 2023-07-29T17:11:11.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[23704]: audit 2023-07-29T17:11:09.716934+0000 mon.a (mon.0) 740 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T17:11:11.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[23704]: audit 2023-07-29T17:11:09.718063+0000 mon.a (mon.0) 741 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:11:11.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[23704]: audit 2023-07-29T17:11:09.719724+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T17:11:11.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[23704]: audit 2023-07-29T17:11:09.720354+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:11:11.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[28259]: cluster 2023-07-29T17:11:09.713425+0000 mon.a (mon.0) 738 : cluster [DBG] Standby manager daemon y restarted 2023-07-29T17:11:11.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[28259]: cluster 2023-07-29T17:11:09.713554+0000 mon.a (mon.0) 739 : cluster [DBG] Standby manager daemon y started 2023-07-29T17:11:11.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[28259]: audit 2023-07-29T17:11:09.716934+0000 mon.a (mon.0) 740 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T17:11:11.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[28259]: audit 2023-07-29T17:11:09.718063+0000 mon.a (mon.0) 741 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:11:11.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[28259]: audit 2023-07-29T17:11:09.719724+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T17:11:11.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:10 smithi078 bash[28259]: audit 2023-07-29T17:11:09.720354+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:11:11.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2023-07-29T17:11:11.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2023-07-29T17:11:11.565 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:11 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:11:11.565 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:11 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:11:11.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[23526]: cluster 2023-07-29T17:11:10.254314+0000 mgr.x (mgr.24745) 100 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:11.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[23526]: cluster 2023-07-29T17:11:10.699563+0000 mon.a (mon.0) 744 : cluster [DBG] mgrmap e21: x(active, since 100s), standbys: y 2023-07-29T17:11:11.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2023-07-29T17:11:11.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2023-07-29T17:11:11.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2023-07-29T17:11:11.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2023-07-29T17:11:11.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2023-07-29T17:11:12.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2023-07-29T17:11:12.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2023-07-29T17:11:12.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:11 smithi129 bash[45976]: t=2023-07-29T17:11:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2023-07-29T17:11:12.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:11 smithi078 bash[28259]: cluster 2023-07-29T17:11:10.254314+0000 mgr.x (mgr.24745) 100 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:12.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:11 smithi078 bash[28259]: cluster 2023-07-29T17:11:10.699563+0000 mon.a (mon.0) 744 : cluster [DBG] mgrmap e21: x(active, since 100s), standbys: y 2023-07-29T17:11:12.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:11 smithi078 bash[23704]: cluster 2023-07-29T17:11:10.254314+0000 mgr.x (mgr.24745) 100 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:11:12.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:11 smithi078 bash[23704]: cluster 2023-07-29T17:11:10.699563+0000 mon.a (mon.0) 744 : cluster [DBG] mgrmap e21: x(active, since 100s), standbys: y 2023-07-29T17:11:12.353 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2023-07-29T17:11:12.354 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2023-07-29T17:11:12.354 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2023-07-29T17:11:12.354 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2023-07-29T17:11:12.354 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-29T17:11:12.647 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-29T17:11:12.647 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2023-07-29T17:11:12.647 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2023-07-29T17:11:12.647 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2023-07-29T17:11:12.916 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2023-07-29T17:11:12.916 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2023-07-29T17:11:12.916 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2023-07-29T17:11:12.916 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-29T17:11:12.917 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2023-07-29T17:11:13.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:12 smithi129 bash[45976]: t=2023-07-29T17:11:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2023-07-29T17:11:13.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2023-07-29T17:11:13.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2023-07-29T17:11:13.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2023-07-29T17:11:13.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-29T17:11:13.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2023-07-29T17:11:13.503 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2023-07-29T17:11:13.504 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2023-07-29T17:11:13.763 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[23526]: cluster 2023-07-29T17:11:12.256147+0000 mgr.x (mgr.24745) 101 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:13.763 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2023-07-29T17:11:13.763 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2023-07-29T17:11:13.764 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2023-07-29T17:11:14.048 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2023-07-29T17:11:14.049 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2023-07-29T17:11:14.049 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2023-07-29T17:11:14.049 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:13 smithi129 bash[45976]: t=2023-07-29T17:11:13+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2023-07-29T17:11:14.049 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2023-07-29T17:11:14.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:13 smithi078 bash[28259]: cluster 2023-07-29T17:11:12.256147+0000 mgr.x (mgr.24745) 101 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:14.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:13 smithi078 bash[23704]: cluster 2023-07-29T17:11:12.256147+0000 mgr.x (mgr.24745) 101 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:14.366 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2023-07-29T17:11:14.366 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2023-07-29T17:11:14.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Created default organization" logger=sqlstore 2023-07-29T17:11:14.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing HTTPServer" logger=server 2023-07-29T17:11:14.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2023-07-29T17:11:14.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing PluginManager" logger=server 2023-07-29T17:11:14.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Starting plugin search" logger=plugins 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing HooksService" logger=server 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing RemoteCache" logger=server 2023-07-29T17:11:14.671 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing RenderingService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing AlertEngine" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing QuotaService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing ServerLockService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing LoginService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing SearchService" logger=server 2023-07-29T17:11:14.672 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing TracingService" logger=server 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing CleanUpService" logger=server 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing NotificationService" logger=server 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki 2023-07-29T17:11:14.673 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2023-07-29T17:11:14.674 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2023-07-29T17:11:14.674 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="Initializing Stream Manager" 2023-07-29T17:11:14.674 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:11:14 smithi129 bash[45976]: t=2023-07-29T17:11:14+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2023-07-29T17:11:15.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:15 smithi078 bash[23704]: cluster 2023-07-29T17:11:14.256978+0000 mgr.x (mgr.24745) 102 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:15.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:15 smithi078 bash[28259]: cluster 2023-07-29T17:11:14.256978+0000 mgr.x (mgr.24745) 102 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:16.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:15 smithi129 bash[23526]: cluster 2023-07-29T17:11:14.256978+0000 mgr.x (mgr.24745) 102 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:17 smithi129 bash[23526]: cluster 2023-07-29T17:11:16.258549+0000 mgr.x (mgr.24745) 103 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:18.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:17 smithi078 bash[23704]: cluster 2023-07-29T17:11:16.258549+0000 mgr.x (mgr.24745) 103 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:18.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:17 smithi078 bash[28259]: cluster 2023-07-29T17:11:16.258549+0000 mgr.x (mgr.24745) 103 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:19.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cluster 2023-07-29T17:11:18.259375+0000 mgr.x (mgr.24745) 104 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.545540+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.559614+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.564619+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.566600+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cephadm 2023-07-29T17:11:18.568228+0000 mgr.x (mgr.24745) 105 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cluster 2023-07-29T17:11:18.569727+0000 mgr.x (mgr.24745) 106 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-29T17:11:19.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.575683+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.664451+0000 mon.b (mon.1) 80 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cephadm 2023-07-29T17:11:18.669628+0000 mgr.x (mgr.24745) 107 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cephadm 2023-07-29T17:11:18.670428+0000 mgr.x (mgr.24745) 108 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.673269+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cephadm 2023-07-29T17:11:18.673567+0000 mgr.x (mgr.24745) 109 : cephadm [INF] Failing over to other MGR 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: audit 2023-07-29T17:11:18.674023+0000 mon.b (mon.1) 81 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cluster 2023-07-29T17:11:18.681996+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-29T17:11:19.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cluster 2023-07-29T17:11:18.712730+0000 mon.a (mon.0) 750 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:11:19.818 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[28259]: cluster 2023-07-29T17:11:18.857821+0000 mon.a (mon.0) 751 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:11:19.818 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[43116]: debug 2023-07-29T17:11:19.592+0000 7fd248275700 -1 mgr handle_mgr_map I was active but no longer am 2023-07-29T17:11:19.819 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[43116]: ignoring --setuser ceph since I am not root 2023-07-29T17:11:19.819 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[43116]: ignoring --setgroup ceph since I am not root 2023-07-29T17:11:19.819 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[43116]: debug 2023-07-29T17:11:19.644+0000 7f3dbca57700 1 -- 172.21.15.129:0/1863554527 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564c90db11e0 con 0x564c91b30c00 2023-07-29T17:11:19.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cluster 2023-07-29T17:11:18.259375+0000 mgr.x (mgr.24745) 104 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:19.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.545540+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.559614+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.564619+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.566600+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cephadm 2023-07-29T17:11:18.568228+0000 mgr.x (mgr.24745) 105 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cluster 2023-07-29T17:11:18.569727+0000 mgr.x (mgr.24745) 106 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.575683+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.664451+0000 mon.b (mon.1) 80 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cephadm 2023-07-29T17:11:18.669628+0000 mgr.x (mgr.24745) 107 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cephadm 2023-07-29T17:11:18.670428+0000 mgr.x (mgr.24745) 108 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.673269+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cephadm 2023-07-29T17:11:18.673567+0000 mgr.x (mgr.24745) 109 : cephadm [INF] Failing over to other MGR 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: audit 2023-07-29T17:11:18.674023+0000 mon.b (mon.1) 81 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cluster 2023-07-29T17:11:18.681996+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cluster 2023-07-29T17:11:18.712730+0000 mon.a (mon.0) 750 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:11:19.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:19 smithi078 bash[23704]: cluster 2023-07-29T17:11:18.857821+0000 mon.a (mon.0) 751 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cluster 2023-07-29T17:11:18.259375+0000 mgr.x (mgr.24745) 104 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.545540+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.559614+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.564619+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.566600+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cephadm 2023-07-29T17:11:18.568228+0000 mgr.x (mgr.24745) 105 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cluster 2023-07-29T17:11:18.569727+0000 mgr.x (mgr.24745) 106 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-29T17:11:19.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.575683+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24745 ' entity='mgr.x' 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.664451+0000 mon.b (mon.1) 80 : audit [DBG] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cephadm 2023-07-29T17:11:18.669628+0000 mgr.x (mgr.24745) 107 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cephadm 2023-07-29T17:11:18.670428+0000 mgr.x (mgr.24745) 108 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.673269+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cephadm 2023-07-29T17:11:18.673567+0000 mgr.x (mgr.24745) 109 : cephadm [INF] Failing over to other MGR 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: audit 2023-07-29T17:11:18.674023+0000 mon.b (mon.1) 81 : audit [INF] from='mgr.24745 172.21.15.129:0/2165877375' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-29T17:11:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cluster 2023-07-29T17:11:18.681996+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-29T17:11:19.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cluster 2023-07-29T17:11:18.712730+0000 mon.a (mon.0) 750 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:11:19.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[23526]: cluster 2023-07-29T17:11:18.857821+0000 mon.a (mon.0) 751 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:11:20.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:19 smithi129 bash[43116]: debug 2023-07-29T17:11:19.816+0000 7f3dc6787200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T17:11:20.868 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[43116]: debug 2023-07-29T17:11:20.748+0000 7f3dc6787200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T17:11:20.869 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.587470+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-07-29T17:11:20.869 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cluster 2023-07-29T17:11:19.588263+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e22: y(active, starting, since 0.913518s), standbys: x 2023-07-29T17:11:20.869 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.590395+0000 mon.a (mon.0) 754 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:11:20.869 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.590609+0000 mon.a (mon.0) 755 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:11:20.869 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.590756+0000 mon.a (mon.0) 756 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.593055+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.593301+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.593641+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.593959+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.594225+0000 mon.a (mon.0) 761 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.594468+0000 mon.a (mon.0) 762 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.594715+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:11:20.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.594951+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:11:20.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.595189+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:11:20.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.595454+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:11:20.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.595785+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:11:20.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.595986+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:11:20.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.596484+0000 mon.a (mon.0) 769 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:11:20.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cluster 2023-07-29T17:11:19.639683+0000 mon.a (mon.0) 770 : cluster [INF] Manager daemon y is now available 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cephadm 2023-07-29T17:11:19.668775+0000 mgr.y (mgr.15066) 1 : cephadm [INF] inventory: adjusted host smithi129 addr 'smithi129' -> '172.21.15.129' 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.675255+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.697610+0000 mon.a (mon.0) 772 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.747515+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: audit 2023-07-29T17:11:19.765881+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cephadm 2023-07-29T17:11:20.194974+0000 mgr.y (mgr.15066) 2 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Bus STARTING 2023-07-29T17:11:20.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cephadm 2023-07-29T17:11:20.317199+0000 mgr.y (mgr.15066) 3 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:11:20.873 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cephadm 2023-07-29T17:11:20.419606+0000 mgr.y (mgr.15066) 4 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:11:20.873 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[23526]: cephadm 2023-07-29T17:11:20.420293+0000 mgr.y (mgr.15066) 5 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Bus STARTED 2023-07-29T17:11:20.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.587470+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-07-29T17:11:20.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cluster 2023-07-29T17:11:19.588263+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e22: y(active, starting, since 0.913518s), standbys: x 2023-07-29T17:11:20.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.590395+0000 mon.a (mon.0) 754 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.590609+0000 mon.a (mon.0) 755 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.590756+0000 mon.a (mon.0) 756 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.593055+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.593301+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.593641+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.593959+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.594225+0000 mon.a (mon.0) 761 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:11:20.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.594468+0000 mon.a (mon.0) 762 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.594715+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.594951+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.595189+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.595454+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.595785+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.595986+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:11:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.596484+0000 mon.a (mon.0) 769 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:11:20.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cluster 2023-07-29T17:11:19.639683+0000 mon.a (mon.0) 770 : cluster [INF] Manager daemon y is now available 2023-07-29T17:11:20.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.587470+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.24745 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-07-29T17:11:20.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cluster 2023-07-29T17:11:19.588263+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e22: y(active, starting, since 0.913518s), standbys: x 2023-07-29T17:11:20.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.590395+0000 mon.a (mon.0) 754 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:11:20.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.590609+0000 mon.a (mon.0) 755 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.590756+0000 mon.a (mon.0) 756 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.593055+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.593301+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.593641+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.593959+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.594225+0000 mon.a (mon.0) 761 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:11:20.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.594468+0000 mon.a (mon.0) 762 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.594715+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.594951+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.595189+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.595454+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.595785+0000 mon.a (mon.0) 767 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.595986+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.596484+0000 mon.a (mon.0) 769 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:11:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cluster 2023-07-29T17:11:19.639683+0000 mon.a (mon.0) 770 : cluster [INF] Manager daemon y is now available 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cephadm 2023-07-29T17:11:19.668775+0000 mgr.y (mgr.15066) 1 : cephadm [INF] inventory: adjusted host smithi129 addr 'smithi129' -> '172.21.15.129' 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.675255+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.697610+0000 mon.a (mon.0) 772 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.747515+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: audit 2023-07-29T17:11:19.765881+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cephadm 2023-07-29T17:11:20.194974+0000 mgr.y (mgr.15066) 2 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Bus STARTING 2023-07-29T17:11:20.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cephadm 2023-07-29T17:11:20.317199+0000 mgr.y (mgr.15066) 3 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cephadm 2023-07-29T17:11:20.419606+0000 mgr.y (mgr.15066) 4 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[28259]: cephadm 2023-07-29T17:11:20.420293+0000 mgr.y (mgr.15066) 5 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Bus STARTED 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cephadm 2023-07-29T17:11:19.668775+0000 mgr.y (mgr.15066) 1 : cephadm [INF] inventory: adjusted host smithi129 addr 'smithi129' -> '172.21.15.129' 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.675255+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.697610+0000 mon.a (mon.0) 772 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.747515+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:11:20.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: audit 2023-07-29T17:11:19.765881+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:11:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cephadm 2023-07-29T17:11:20.194974+0000 mgr.y (mgr.15066) 2 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Bus STARTING 2023-07-29T17:11:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cephadm 2023-07-29T17:11:20.317199+0000 mgr.y (mgr.15066) 3 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:11:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cephadm 2023-07-29T17:11:20.419606+0000 mgr.y (mgr.15066) 4 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:11:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:20 smithi078 bash[23704]: cephadm 2023-07-29T17:11:20.420293+0000 mgr.y (mgr.15066) 5 : cephadm [INF] [29/Jul/2023:17:11:20] ENGINE Bus STARTED 2023-07-29T17:11:21.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[43116]: debug 2023-07-29T17:11:20.848+0000 7f3dc6787200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T17:11:21.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:20 smithi129 bash[43116]: debug 2023-07-29T17:11:20.948+0000 7f3dc6787200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T17:11:21.579 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:21 smithi129 bash[43116]: debug 2023-07-29T17:11:21.208+0000 7f3dc6787200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:11:21.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:21 smithi129 bash[43116]: debug 2023-07-29T17:11:21.560+0000 7f3dc6787200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T17:11:21.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:21 smithi129 bash[43116]: debug 2023-07-29T17:11:21.724+0000 7f3dc6787200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T17:11:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:21 smithi129 bash[23526]: cluster 2023-07-29T17:11:20.602130+0000 mon.a (mon.0) 775 : cluster [DBG] mgrmap e23: y(active, since 1.92739s), standbys: x 2023-07-29T17:11:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:21 smithi129 bash[23526]: cluster 2023-07-29T17:11:20.640379+0000 mgr.y (mgr.15066) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:21.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:21 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2023-07-29T17:11:21.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:21 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:21.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:21 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:21.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:21 smithi078 bash[23704]: cluster 2023-07-29T17:11:20.602130+0000 mon.a (mon.0) 775 : cluster [DBG] mgrmap e23: y(active, since 1.92739s), standbys: x 2023-07-29T17:11:21.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:21 smithi078 bash[23704]: cluster 2023-07-29T17:11:20.640379+0000 mgr.y (mgr.15066) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:21.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:21 smithi078 bash[28259]: cluster 2023-07-29T17:11:20.602130+0000 mon.a (mon.0) 775 : cluster [DBG] mgrmap e23: y(active, since 1.92739s), standbys: x 2023-07-29T17:11:21.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:21 smithi078 bash[28259]: cluster 2023-07-29T17:11:20.640379+0000 mgr.y (mgr.15066) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:22.174 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:21 smithi129 bash[43116]: debug 2023-07-29T17:11:21.944+0000 7f3dc6787200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T17:11:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:22 smithi129 bash[23526]: cluster 2023-07-29T17:11:21.594817+0000 mon.a (mon.0) 776 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-29T17:11:22.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:22 smithi129 bash[23526]: cluster 2023-07-29T17:11:21.594912+0000 mon.a (mon.0) 777 : cluster [INF] Cluster is now healthy 2023-07-29T17:11:22.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:22 smithi129 bash[23526]: cluster 2023-07-29T17:11:21.595385+0000 mgr.y (mgr.15066) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:22.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:22 smithi078 bash[23704]: cluster 2023-07-29T17:11:21.594817+0000 mon.a (mon.0) 776 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-29T17:11:22.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:22 smithi078 bash[23704]: cluster 2023-07-29T17:11:21.594912+0000 mon.a (mon.0) 777 : cluster [INF] Cluster is now healthy 2023-07-29T17:11:22.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:22 smithi078 bash[23704]: cluster 2023-07-29T17:11:21.595385+0000 mgr.y (mgr.15066) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:22.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:22 smithi078 bash[28259]: cluster 2023-07-29T17:11:21.594817+0000 mon.a (mon.0) 776 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-29T17:11:22.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:22 smithi078 bash[28259]: cluster 2023-07-29T17:11:21.594912+0000 mon.a (mon.0) 777 : cluster [INF] Cluster is now healthy 2023-07-29T17:11:22.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:22 smithi078 bash[28259]: cluster 2023-07-29T17:11:21.595385+0000 mgr.y (mgr.15066) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:23.239 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:22 smithi129 bash[43116]: debug 2023-07-29T17:11:22.964+0000 7f3dc6787200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T17:11:23.239 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[43116]: debug 2023-07-29T17:11:23.088+0000 7f3dc6787200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T17:11:23.239 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.097Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:11:23.240 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.097Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:11:23.240 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.097Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:11:23.240 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.097Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:11:23.240 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.097Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:11:23.240 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.097Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:11:23.241 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.099Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:11:23.241 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.099Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:11:23.241 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.102Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:11:23.241 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.103Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2023-07-29T17:11:23.241 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.116Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2023-07-29T17:11:23.241 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.116Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.116Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.116Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.117Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.117Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.117Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=14.641494ms 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.119Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:11:23.242 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.119Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.119Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:11:23.243 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:11:23.244 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:11:23.244 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:11:23.244 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:11:23.244 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=info ts=2023-07-29T17:11:23.120Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:11:23.244 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[47408]: level=error ts=2023-07-29T17:11:23.120Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:11:23.649 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[43116]: debug 2023-07-29T17:11:23.220+0000 7f3dc6787200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T17:11:23.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[23526]: cluster 2023-07-29T17:11:22.619520+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e24: y(active, since 3s), standbys: x 2023-07-29T17:11:23.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[43116]: debug 2023-07-29T17:11:23.712+0000 7f3dc6787200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T17:11:23.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:11:23.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:23 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:11:23.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:23 smithi078 bash[23704]: cluster 2023-07-29T17:11:22.619520+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e24: y(active, since 3s), standbys: x 2023-07-29T17:11:23.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:23 smithi078 bash[28259]: cluster 2023-07-29T17:11:22.619520+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e24: y(active, since 3s), standbys: x 2023-07-29T17:11:24.329 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:23 smithi129 bash[43116]: debug 2023-07-29T17:11:23.936+0000 7f3dc6787200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T17:11:24.652 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:24 smithi129 bash[43116]: debug 2023-07-29T17:11:24.308+0000 7f3dc6787200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T17:11:24.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:24 smithi129 bash[43116]: debug 2023-07-29T17:11:24.800+0000 7f3dc6787200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:11:24.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:24 smithi129 bash[23526]: cluster 2023-07-29T17:11:23.596236+0000 mgr.y (mgr.15066) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:24.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:24 smithi078 bash[23704]: cluster 2023-07-29T17:11:23.596236+0000 mgr.y (mgr.15066) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:24.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:24 smithi078 bash[28259]: cluster 2023-07-29T17:11:23.596236+0000 mgr.y (mgr.15066) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:25.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:25 smithi129 bash[43116]: debug 2023-07-29T17:11:25.364+0000 7f3dc6787200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T17:11:25.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:25 smithi129 bash[43116]: debug 2023-07-29T17:11:25.476+0000 7f3dc6787200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T17:11:26.154 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:25 smithi129 bash[43116]: debug 2023-07-29T17:11:25.864+0000 7f3dc6787200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T17:11:26.154 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:25 smithi129 bash[43116]: debug 2023-07-29T17:11:25.984+0000 7f3dc6787200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T17:11:26.413 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:26 smithi129 bash[43116]: debug 2023-07-29T17:11:26.136+0000 7f3dc6787200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T17:11:26.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:26 smithi129 bash[43116]: debug 2023-07-29T17:11:26.252+0000 7f3dc6787200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T17:11:26.682 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:26 smithi129 bash[23526]: cluster 2023-07-29T17:11:25.596864+0000 mgr.y (mgr.15066) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:26.682 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:26 smithi129 bash[43116]: debug 2023-07-29T17:11:26.392+0000 7f3dc6787200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T17:11:26.683 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:26 smithi129 bash[43116]: debug 2023-07-29T17:11:26.612+0000 7f3dc6787200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T17:11:26.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:26 smithi078 bash[28259]: cluster 2023-07-29T17:11:25.596864+0000 mgr.y (mgr.15066) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:26.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:26 smithi078 bash[23704]: cluster 2023-07-29T17:11:25.596864+0000 mgr.y (mgr.15066) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:11:27.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:26 smithi129 bash[43116]: debug 2023-07-29T17:11:26.740+0000 7f3dc6787200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T17:11:27.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[28259]: audit 2023-07-29T17:11:26.652727+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[28259]: audit 2023-07-29T17:11:26.661098+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[28259]: audit 2023-07-29T17:11:27.197452+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[28259]: audit 2023-07-29T17:11:27.207710+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[28259]: audit 2023-07-29T17:11:27.456822+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[28259]: audit 2023-07-29T17:11:27.469386+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[23704]: audit 2023-07-29T17:11:26.652727+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[23704]: audit 2023-07-29T17:11:26.661098+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[23704]: audit 2023-07-29T17:11:27.197452+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[23704]: audit 2023-07-29T17:11:27.207710+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[23704]: audit 2023-07-29T17:11:27.456822+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:27.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:27 smithi078 bash[23704]: audit 2023-07-29T17:11:27.469386+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.030 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:27 smithi129 bash[23526]: audit 2023-07-29T17:11:26.652727+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.031 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:27 smithi129 bash[23526]: audit 2023-07-29T17:11:26.661098+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.031 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:27 smithi129 bash[23526]: audit 2023-07-29T17:11:27.197452+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.031 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:27 smithi129 bash[23526]: audit 2023-07-29T17:11:27.207710+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.031 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:27 smithi129 bash[23526]: audit 2023-07-29T17:11:27.456822+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.031 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:27 smithi129 bash[23526]: audit 2023-07-29T17:11:27.469386+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:28.420 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[43116]: debug 2023-07-29T17:11:28.012+0000 7f3dc6787200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T17:11:29.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: cluster 2023-07-29T17:11:27.597897+0000 mgr.y (mgr.15066) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2023-07-29T17:11:29.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: audit 2023-07-29T17:11:27.959581+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:29.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: audit 2023-07-29T17:11:27.967858+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:29.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: cluster 2023-07-29T17:11:28.016655+0000 mon.a (mon.0) 787 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:11:29.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: cluster 2023-07-29T17:11:28.016833+0000 mon.a (mon.0) 788 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:11:29.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: audit 2023-07-29T17:11:28.021107+0000 mon.b (mon.1) 82 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:11:29.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: audit 2023-07-29T17:11:28.025293+0000 mon.b (mon.1) 83 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:11:29.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: audit 2023-07-29T17:11:28.027199+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:11:29.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:28 smithi129 bash[23526]: audit 2023-07-29T17:11:28.027805+0000 mon.b (mon.1) 85 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:11:29.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: cluster 2023-07-29T17:11:27.597897+0000 mgr.y (mgr.15066) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2023-07-29T17:11:29.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: audit 2023-07-29T17:11:27.959581+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:29.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: audit 2023-07-29T17:11:27.967858+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:29.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: cluster 2023-07-29T17:11:28.016655+0000 mon.a (mon.0) 787 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:11:29.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: cluster 2023-07-29T17:11:28.016833+0000 mon.a (mon.0) 788 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:11:29.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: audit 2023-07-29T17:11:28.021107+0000 mon.b (mon.1) 82 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:11:29.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: audit 2023-07-29T17:11:28.025293+0000 mon.b (mon.1) 83 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:11:29.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: audit 2023-07-29T17:11:28.027199+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:11:29.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[23704]: audit 2023-07-29T17:11:28.027805+0000 mon.b (mon.1) 85 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:11:29.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: cluster 2023-07-29T17:11:27.597897+0000 mgr.y (mgr.15066) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2023-07-29T17:11:29.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: audit 2023-07-29T17:11:27.959581+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: audit 2023-07-29T17:11:27.967858+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: cluster 2023-07-29T17:11:28.016655+0000 mon.a (mon.0) 787 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: cluster 2023-07-29T17:11:28.016833+0000 mon.a (mon.0) 788 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: audit 2023-07-29T17:11:28.021107+0000 mon.b (mon.1) 82 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: audit 2023-07-29T17:11:28.025293+0000 mon.b (mon.1) 83 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: audit 2023-07-29T17:11:28.027199+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:11:29.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:28 smithi078 bash[28259]: audit 2023-07-29T17:11:28.027805+0000 mon.b (mon.1) 85 : audit [DBG] from='mgr.? 172.21.15.129:0/575989077' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:11:30.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:29 smithi129 bash[23526]: cluster 2023-07-29T17:11:28.981534+0000 mon.a (mon.0) 789 : cluster [DBG] mgrmap e25: y(active, since 10s), standbys: x 2023-07-29T17:11:30.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:29 smithi078 bash[23704]: cluster 2023-07-29T17:11:28.981534+0000 mon.a (mon.0) 789 : cluster [DBG] mgrmap e25: y(active, since 10s), standbys: x 2023-07-29T17:11:30.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:29 smithi078 bash[28259]: cluster 2023-07-29T17:11:28.981534+0000 mon.a (mon.0) 789 : cluster [DBG] mgrmap e25: y(active, since 10s), standbys: x 2023-07-29T17:11:30.929 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:11:31.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:30 smithi129 bash[23526]: cluster 2023-07-29T17:11:29.598653+0000 mgr.y (mgr.15066) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2023-07-29T17:11:31.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:30 smithi078 bash[23704]: cluster 2023-07-29T17:11:29.598653+0000 mgr.y (mgr.15066) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2023-07-29T17:11:31.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:30 smithi078 bash[28259]: cluster 2023-07-29T17:11:29.598653+0000 mgr.y (mgr.15066) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2023-07-29T17:11:31.674 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:11:31.674 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (87s) 5s ago 7m 10.1M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:11:31.674 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (44s) 4s ago 7m 22.6M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:11:31.674 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (37s) 4s ago 5m 43.0M - 3.4 24ecd6d5f14c e9389cf80a9c 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443 running (2m) 4s ago 11m 152M - 18.1.3-11-g0484097b b662b4d80c86 1ad497f13c14 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (31s) 5s ago 13m 456M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (13m) 5s ago 13m 60.0M 2048M 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (12m) 4s ago 12m 52.2M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (12m) 5s ago 12m 48.9M 2048M 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (96s) 5s ago 7m 8880k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (70s) 4s ago 7m 3307k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:11:31.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (11m) 5s ago 11m 55.0M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (11m) 5s ago 11m 52.6M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (10m) 5s ago 10m 47.3M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10m) 5s ago 10m 52.9M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (9m) 4s ago 9m 54.1M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (9m) 4s ago 9m 51.8M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (9m) 4s ago 9m 52.0M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:11:31.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (8m) 4s ago 8m 55.2M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:11:31.677 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 4s ago 7m - - 2023-07-29T17:11:31.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (5m) 5s ago 5m 53.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:11:31.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (5m) 4s ago 5m 53.2M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:11:31.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (5m) 5s ago 5m 53.1M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:11:31.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (5m) 4s ago 5m 53.4M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:11:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:11:32.110 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:11:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2023-07-29T17:11:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:11:32.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:11:32.113 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:11:32.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:31 smithi129 bash[23526]: audit 2023-07-29T17:11:30.897689+0000 mgr.y (mgr.15066) 12 : audit [DBG] from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:32.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:31 smithi129 bash[23526]: audit 2023-07-29T17:11:31.270725+0000 mgr.y (mgr.15066) 13 : audit [DBG] from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:32.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:31 smithi078 bash[23704]: audit 2023-07-29T17:11:30.897689+0000 mgr.y (mgr.15066) 12 : audit [DBG] from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:32.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:31 smithi078 bash[23704]: audit 2023-07-29T17:11:31.270725+0000 mgr.y (mgr.15066) 13 : audit [DBG] from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:32.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:31 smithi078 bash[28259]: audit 2023-07-29T17:11:30.897689+0000 mgr.y (mgr.15066) 12 : audit [DBG] from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:32.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:31 smithi078 bash[28259]: audit 2023-07-29T17:11:31.270725+0000 mgr.y (mgr.15066) 13 : audit [DBG] from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:32.487 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:11:32.487 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:11:32.487 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:11:32.487 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:11:32.487 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:11:32.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2023-07-29T17:11:32.488 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:11:32.488 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "2/23 daemons upgraded", 2023-07-29T17:11:32.488 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "", 2023-07-29T17:11:32.488 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:11:32.488 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:11:32.913 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:11:33.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:32 smithi129 bash[23526]: cluster 2023-07-29T17:11:31.600300+0000 mgr.y (mgr.15066) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2023-07-29T17:11:33.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:32 smithi129 bash[23526]: audit 2023-07-29T17:11:31.639873+0000 mgr.y (mgr.15066) 15 : audit [DBG] from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:33.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:32 smithi129 bash[23526]: audit 2023-07-29T17:11:32.093412+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? 172.21.15.78:0/4038366633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:11:33.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:32 smithi129 bash[23526]: audit 2023-07-29T17:11:32.469610+0000 mgr.y (mgr.15066) 16 : audit [DBG] from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:33.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:32 smithi129 bash[23526]: audit 2023-07-29T17:11:32.896631+0000 mon.a (mon.0) 790 : audit [DBG] from='client.? 172.21.15.78:0/2632951924' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:11:33.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[28259]: cluster 2023-07-29T17:11:31.600300+0000 mgr.y (mgr.15066) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2023-07-29T17:11:33.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[28259]: audit 2023-07-29T17:11:31.639873+0000 mgr.y (mgr.15066) 15 : audit [DBG] from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:33.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[28259]: audit 2023-07-29T17:11:32.093412+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? 172.21.15.78:0/4038366633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:11:33.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[28259]: audit 2023-07-29T17:11:32.469610+0000 mgr.y (mgr.15066) 16 : audit [DBG] from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:33.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[28259]: audit 2023-07-29T17:11:32.896631+0000 mon.a (mon.0) 790 : audit [DBG] from='client.? 172.21.15.78:0/2632951924' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:11:33.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[23704]: cluster 2023-07-29T17:11:31.600300+0000 mgr.y (mgr.15066) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2023-07-29T17:11:33.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[23704]: audit 2023-07-29T17:11:31.639873+0000 mgr.y (mgr.15066) 15 : audit [DBG] from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:33.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[23704]: audit 2023-07-29T17:11:32.093412+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? 172.21.15.78:0/4038366633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:11:33.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[23704]: audit 2023-07-29T17:11:32.469610+0000 mgr.y (mgr.15066) 16 : audit [DBG] from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:11:33.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:32 smithi078 bash[23704]: audit 2023-07-29T17:11:32.896631+0000 mon.a (mon.0) 790 : audit [DBG] from='client.? 172.21.15.78:0/2632951924' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:11:34.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:33 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 5. 2023-07-29T17:11:34.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:33 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:34.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:33 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Start request repeated too quickly. 2023-07-29T17:11:34.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:33 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:11:34.421 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:33 smithi129 systemd[1]: Failed to start Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:35.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:34 smithi078 bash[23704]: cluster 2023-07-29T17:11:33.601227+0000 mgr.y (mgr.15066) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:35.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:35 smithi078 bash[28259]: cluster 2023-07-29T17:11:33.601227+0000 mgr.y (mgr.15066) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:35.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:35 smithi129 bash[23526]: cluster 2023-07-29T17:11:33.601227+0000 mgr.y (mgr.15066) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:37.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:37 smithi078 bash[23704]: cluster 2023-07-29T17:11:35.602094+0000 mgr.y (mgr.15066) 18 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:37.328 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:37 smithi078 bash[28259]: cluster 2023-07-29T17:11:35.602094+0000 mgr.y (mgr.15066) 18 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:37.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:37 smithi129 bash[23526]: cluster 2023-07-29T17:11:35.602094+0000 mgr.y (mgr.15066) 18 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:39.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: cluster 2023-07-29T17:11:37.603847+0000 mgr.y (mgr.15066) 19 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:39.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: audit 2023-07-29T17:11:37.886150+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: audit 2023-07-29T17:11:37.896508+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: audit 2023-07-29T17:11:37.898638+0000 mon.a (mon.0) 793 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: audit 2023-07-29T17:11:37.899913+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: cephadm 2023-07-29T17:11:37.901163+0000 mgr.y (mgr.15066) 20 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: cluster 2023-07-29T17:11:37.902260+0000 mgr.y (mgr.15066) 21 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: audit 2023-07-29T17:11:37.909917+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: cephadm 2023-07-29T17:11:37.984920+0000 mgr.y (mgr.15066) 22 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: cluster 2023-07-29T17:11:38.004399+0000 mon.a (mon.0) 796 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:11:39.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:38 smithi129 bash[23526]: cephadm 2023-07-29T17:11:38.232589+0000 mgr.y (mgr.15066) 23 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:11:39.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: cluster 2023-07-29T17:11:37.603847+0000 mgr.y (mgr.15066) 19 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:39.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: audit 2023-07-29T17:11:37.886150+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: audit 2023-07-29T17:11:37.896508+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: audit 2023-07-29T17:11:37.898638+0000 mon.a (mon.0) 793 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:39.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: audit 2023-07-29T17:11:37.899913+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:11:39.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: cephadm 2023-07-29T17:11:37.901163+0000 mgr.y (mgr.15066) 20 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:11:39.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: cluster 2023-07-29T17:11:37.902260+0000 mgr.y (mgr.15066) 21 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:39.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: audit 2023-07-29T17:11:37.909917+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: cephadm 2023-07-29T17:11:37.984920+0000 mgr.y (mgr.15066) 22 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:11:39.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: cluster 2023-07-29T17:11:38.004399+0000 mon.a (mon.0) 796 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:11:39.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[23704]: cephadm 2023-07-29T17:11:38.232589+0000 mgr.y (mgr.15066) 23 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:11:39.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: cluster 2023-07-29T17:11:37.603847+0000 mgr.y (mgr.15066) 19 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:11:39.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: audit 2023-07-29T17:11:37.886150+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: audit 2023-07-29T17:11:37.896508+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: audit 2023-07-29T17:11:37.898638+0000 mon.a (mon.0) 793 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: audit 2023-07-29T17:11:37.899913+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: cephadm 2023-07-29T17:11:37.901163+0000 mgr.y (mgr.15066) 20 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: cluster 2023-07-29T17:11:37.902260+0000 mgr.y (mgr.15066) 21 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: audit 2023-07-29T17:11:37.909917+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: cephadm 2023-07-29T17:11:37.984920+0000 mgr.y (mgr.15066) 22 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: cluster 2023-07-29T17:11:38.004399+0000 mon.a (mon.0) 796 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:11:39.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:38 smithi078 bash[28259]: cephadm 2023-07-29T17:11:38.232589+0000 mgr.y (mgr.15066) 23 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:11:41.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:40 smithi129 bash[23526]: cluster 2023-07-29T17:11:39.902979+0000 mgr.y (mgr.15066) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:41.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:40 smithi078 bash[23704]: cluster 2023-07-29T17:11:39.902979+0000 mgr.y (mgr.15066) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:41.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:40 smithi078 bash[28259]: cluster 2023-07-29T17:11:39.902979+0000 mgr.y (mgr.15066) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:42.838 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:42 smithi129 bash[23526]: cluster 2023-07-29T17:11:41.904635+0000 mgr.y (mgr.15066) 25 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:42.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:42 smithi078 bash[23704]: cluster 2023-07-29T17:11:41.904635+0000 mgr.y (mgr.15066) 25 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:42.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:42 smithi078 bash[28259]: cluster 2023-07-29T17:11:41.904635+0000 mgr.y (mgr.15066) 25 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:43.170 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:42 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:44.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:43 smithi129 bash[23526]: audit 2023-07-29T17:11:42.869158+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:44.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:43 smithi129 bash[23526]: audit 2023-07-29T17:11:42.876884+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:44.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:43 smithi129 bash[23526]: cephadm 2023-07-29T17:11:42.914135+0000 mgr.y (mgr.15066) 26 : cephadm [INF] Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:11:44.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:43 smithi129 bash[23526]: audit 2023-07-29T17:11:42.914839+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:11:44.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:43 smithi129 bash[23526]: audit 2023-07-29T17:11:42.927595+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:44.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:43 smithi129 bash[23526]: cephadm 2023-07-29T17:11:42.929361+0000 mgr.y (mgr.15066) 27 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:11:44.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[23704]: audit 2023-07-29T17:11:42.869158+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:44.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[23704]: audit 2023-07-29T17:11:42.876884+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:44.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[23704]: cephadm 2023-07-29T17:11:42.914135+0000 mgr.y (mgr.15066) 26 : cephadm [INF] Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:11:44.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[23704]: audit 2023-07-29T17:11:42.914839+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:11:44.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[23704]: audit 2023-07-29T17:11:42.927595+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:44.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[23704]: cephadm 2023-07-29T17:11:42.929361+0000 mgr.y (mgr.15066) 27 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:11:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[28259]: audit 2023-07-29T17:11:42.869158+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[28259]: audit 2023-07-29T17:11:42.876884+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[28259]: cephadm 2023-07-29T17:11:42.914135+0000 mgr.y (mgr.15066) 26 : cephadm [INF] Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:11:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[28259]: audit 2023-07-29T17:11:42.914839+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:11:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[28259]: audit 2023-07-29T17:11:42.927595+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:43 smithi078 bash[28259]: cephadm 2023-07-29T17:11:42.929361+0000 mgr.y (mgr.15066) 27 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:11:44.815 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.686Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:11:44.815 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.687Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:11:44.815 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.687Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:11:44.815 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.687Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:11:44.815 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.687Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:11:44.816 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.687Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:11:44.816 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.691Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:11:44.816 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.691Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:11:44.816 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.695Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:11:44.816 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.695Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=7 2023-07-29T17:11:44.816 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.708Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.708Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.708Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.708Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.708Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.708Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.709Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=7 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.709Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=14.242606ms 2023-07-29T17:11:44.817 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.711Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.711Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.711Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:11:44.818 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:11:44.819 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:11:44.819 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:11:44.819 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:11:44.819 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:11:44.819 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=info ts=2023-07-29T17:11:44.712Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:11:44.819 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[48427]: level=error ts=2023-07-29T17:11:44.712Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:11:45.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:44 smithi129 bash[23526]: cluster 2023-07-29T17:11:43.905398+0000 mgr.y (mgr.15066) 28 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:45.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:44 smithi078 bash[23704]: cluster 2023-07-29T17:11:43.905398+0000 mgr.y (mgr.15066) 28 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:45.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:44 smithi078 bash[28259]: cluster 2023-07-29T17:11:43.905398+0000 mgr.y (mgr.15066) 28 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:46.146 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:45 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:11:46.146 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:45 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:11:47.132 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:46 smithi129 bash[23526]: cluster 2023-07-29T17:11:45.907027+0000 mgr.y (mgr.15066) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-29T17:11:47.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:46 smithi078 bash[23704]: cluster 2023-07-29T17:11:45.907027+0000 mgr.y (mgr.15066) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-29T17:11:47.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:46 smithi078 bash[28259]: cluster 2023-07-29T17:11:45.907027+0000 mgr.y (mgr.15066) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-29T17:11:49.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.764524+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.772744+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.815780+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:11:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.816830+0000 mgr.y (mgr.15066) 30 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:11:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.819410+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:11:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.820203+0000 mgr.y (mgr.15066) 31 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:11:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.830522+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.841865+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:11:49.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.843111+0000 mgr.y (mgr.15066) 32 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:11:49.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.859371+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:11:49.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.860189+0000 mgr.y (mgr.15066) 33 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:11:49.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: audit 2023-07-29T17:11:47.874487+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:49.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:48 smithi129 bash[23526]: cluster 2023-07-29T17:11:47.907731+0000 mgr.y (mgr.15066) 34 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:49.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.764524+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.772744+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.815780+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.816830+0000 mgr.y (mgr.15066) 30 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.819410+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.820203+0000 mgr.y (mgr.15066) 31 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.830522+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.841865+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.843111+0000 mgr.y (mgr.15066) 32 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:11:49.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.859371+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:11:49.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.860189+0000 mgr.y (mgr.15066) 33 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:11:49.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: audit 2023-07-29T17:11:47.874487+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:49.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[23704]: cluster 2023-07-29T17:11:47.907731+0000 mgr.y (mgr.15066) 34 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:49.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.764524+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.772744+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.815780+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.816830+0000 mgr.y (mgr.15066) 30 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.819410+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.820203+0000 mgr.y (mgr.15066) 31 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129:9095"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.830522+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.841865+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.843111+0000 mgr.y (mgr.15066) 32 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.859371+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:11:49.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.860189+0000 mgr.y (mgr.15066) 33 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:11:49.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: audit 2023-07-29T17:11:47.874487+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:11:49.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:48 smithi078 bash[28259]: cluster 2023-07-29T17:11:47.907731+0000 mgr.y (mgr.15066) 34 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-29T17:11:51.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:50 smithi129 bash[23526]: cluster 2023-07-29T17:11:49.908426+0000 mgr.y (mgr.15066) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:51.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:50 smithi129 bash[23526]: audit 2023-07-29T17:11:50.849688+0000 mon.b (mon.1) 86 : audit [DBG] from='client.? 172.21.15.129:0/2372527162' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:11:51.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:50 smithi078 bash[23704]: cluster 2023-07-29T17:11:49.908426+0000 mgr.y (mgr.15066) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:51.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:50 smithi078 bash[23704]: audit 2023-07-29T17:11:50.849688+0000 mon.b (mon.1) 86 : audit [DBG] from='client.? 172.21.15.129:0/2372527162' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:11:51.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:50 smithi078 bash[28259]: cluster 2023-07-29T17:11:49.908426+0000 mgr.y (mgr.15066) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:11:51.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:50 smithi078 bash[28259]: audit 2023-07-29T17:11:50.849688+0000 mon.b (mon.1) 86 : audit [DBG] from='client.? 172.21.15.129:0/2372527162' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:11:52.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:51 smithi129 bash[23526]: cephadm 2023-07-29T17:11:51.152326+0000 mgr.y (mgr.15066) 36 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-29T17:11:52.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:51 smithi129 bash[23526]: audit 2023-07-29T17:11:51.153159+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:11:52.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:51 smithi129 bash[23526]: audit 2023-07-29T17:11:51.155286+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:11:52.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:51 smithi129 bash[23526]: audit 2023-07-29T17:11:51.157218+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:52.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:51 smithi129 bash[23526]: cephadm 2023-07-29T17:11:51.159128+0000 mgr.y (mgr.15066) 37 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T17:11:52.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:51 smithi129 bash[23526]: audit 2023-07-29T17:11:51.219285+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.129:0/4437790' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:11:52.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[23704]: cephadm 2023-07-29T17:11:51.152326+0000 mgr.y (mgr.15066) 36 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-29T17:11:52.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[23704]: audit 2023-07-29T17:11:51.153159+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:11:52.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[23704]: audit 2023-07-29T17:11:51.155286+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:11:52.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[23704]: audit 2023-07-29T17:11:51.157218+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:52.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[23704]: cephadm 2023-07-29T17:11:51.159128+0000 mgr.y (mgr.15066) 37 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T17:11:52.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[23704]: audit 2023-07-29T17:11:51.219285+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.129:0/4437790' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:11:52.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[28259]: cephadm 2023-07-29T17:11:51.152326+0000 mgr.y (mgr.15066) 36 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-29T17:11:52.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[28259]: audit 2023-07-29T17:11:51.153159+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:11:52.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[28259]: audit 2023-07-29T17:11:51.155286+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:11:52.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[28259]: audit 2023-07-29T17:11:51.157218+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:11:52.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[28259]: cephadm 2023-07-29T17:11:51.159128+0000 mgr.y (mgr.15066) 37 : cephadm [INF] Deploying daemon mgr.x on smithi129 2023-07-29T17:11:52.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:51 smithi078 bash[28259]: audit 2023-07-29T17:11:51.219285+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.129:0/4437790' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:11:53.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:52 smithi129 bash[23526]: cluster 2023-07-29T17:11:51.910118+0000 mgr.y (mgr.15066) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:11:53.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:52 smithi078 bash[23704]: cluster 2023-07-29T17:11:51.910118+0000 mgr.y (mgr.15066) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:11:53.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:52 smithi078 bash[28259]: cluster 2023-07-29T17:11:51.910118+0000 mgr.y (mgr.15066) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:11:54.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:54 smithi129 systemd[1]: Stopping Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:11:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:54 smithi129 bash[23526]: cluster 2023-07-29T17:11:53.910932+0000 mgr.y (mgr.15066) 39 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-07-29T17:11:55.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:54 smithi078 bash[23704]: cluster 2023-07-29T17:11:53.910932+0000 mgr.y (mgr.15066) 39 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-07-29T17:11:55.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:54 smithi078 bash[28259]: cluster 2023-07-29T17:11:53.910932+0000 mgr.y (mgr.15066) 39 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-07-29T17:11:55.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:55 smithi129 bash[49187]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mgr.x 2023-07-29T17:11:55.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:11:55.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2023-07-29T17:11:55.921 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: Stopped Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:56.451 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: Started Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:56.452 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2023-07-29T17:11:56.452 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:56.452 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:55 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:11:57.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[23526]: cluster 2023-07-29T17:11:55.912571+0000 mgr.y (mgr.15066) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2023-07-29T17:11:57.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[23526]: audit 2023-07-29T17:11:56.015637+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:57.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[23526]: audit 2023-07-29T17:11:56.025132+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:57.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:57 smithi078 bash[23704]: cluster 2023-07-29T17:11:55.912571+0000 mgr.y (mgr.15066) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2023-07-29T17:11:57.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:57 smithi078 bash[23704]: audit 2023-07-29T17:11:56.015637+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:57.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:57 smithi078 bash[23704]: audit 2023-07-29T17:11:56.025132+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:57.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:57 smithi078 bash[28259]: cluster 2023-07-29T17:11:55.912571+0000 mgr.y (mgr.15066) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2023-07-29T17:11:57.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:57 smithi078 bash[28259]: audit 2023-07-29T17:11:56.015637+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:57.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:57 smithi078 bash[28259]: audit 2023-07-29T17:11:56.025132+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:11:57.884 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49373]: debug 2023-07-29T17:11:57.691+0000 7f296da6e700 1 -- 172.21.15.129:0/3221275324 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5646b43b11e0 con 0x5646b5130400 2023-07-29T17:11:57.885 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.716Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:11:57.885 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.717Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:11:57.885 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.717Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:11:57.885 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.717Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:11:57.885 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.717Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:11:57.885 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.717Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.720Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.720Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.724Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.724Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=8 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.738Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=8 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.739Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=8 2023-07-29T17:11:57.886 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.739Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=8 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.739Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=8 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.739Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=8 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.739Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=8 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.740Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=8 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.740Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=8 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.740Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=16.374294ms 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.744Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:11:57.887 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.744Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.744Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:11:57.888 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:11:57.889 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:11:57.889 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:11:57.889 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:11:57.889 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=info ts=2023-07-29T17:11:57.745Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:11:57.889 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49353]: level=error ts=2023-07-29T17:11:57.745Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:11:58.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:57 smithi129 bash[49373]: debug 2023-07-29T17:11:57.863+0000 7f2977bbc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T17:11:58.890 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:58 smithi129 bash[49373]: debug 2023-07-29T17:11:58.767+0000 7f2977bbc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T17:11:58.890 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:58 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:11:58.891 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:11:58 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:11:59.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:11:59 smithi129 bash[23526]: cluster 2023-07-29T17:11:57.913375+0000 mgr.y (mgr.15066) 41 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:11:59.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:58 smithi129 bash[49373]: debug 2023-07-29T17:11:58.867+0000 7f2977bbc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T17:11:59.171 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:58 smithi129 bash[49373]: debug 2023-07-29T17:11:58.967+0000 7f2977bbc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T17:11:59.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:11:59 smithi078 bash[23704]: cluster 2023-07-29T17:11:57.913375+0000 mgr.y (mgr.15066) 41 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:11:59.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:11:59 smithi078 bash[28259]: cluster 2023-07-29T17:11:57.913375+0000 mgr.y (mgr.15066) 41 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:11:59.599 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:59 smithi129 bash[49373]: debug 2023-07-29T17:11:59.227+0000 7f2977bbc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:11:59.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:59 smithi129 bash[49373]: debug 2023-07-29T17:11:59.579+0000 7f2977bbc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T17:11:59.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:59 smithi129 bash[49373]: debug 2023-07-29T17:11:59.739+0000 7f2977bbc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T17:12:00.420 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:11:59 smithi129 bash[49373]: debug 2023-07-29T17:11:59.947+0000 7f2977bbc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T17:12:01.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:01 smithi129 bash[23526]: cluster 2023-07-29T17:11:59.914181+0000 mgr.y (mgr.15066) 42 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:12:01.420 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:00 smithi129 bash[49373]: debug 2023-07-29T17:12:00.959+0000 7f2977bbc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T17:12:01.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:01 smithi129 bash[49373]: debug 2023-07-29T17:12:01.075+0000 7f2977bbc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T17:12:01.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:01 smithi129 bash[49373]: debug 2023-07-29T17:12:01.199+0000 7f2977bbc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T17:12:01.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:01 smithi078 bash[23704]: cluster 2023-07-29T17:11:59.914181+0000 mgr.y (mgr.15066) 42 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:12:01.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:01 smithi078 bash[28259]: cluster 2023-07-29T17:11:59.914181+0000 mgr.y (mgr.15066) 42 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:12:02.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:01 smithi129 bash[49373]: debug 2023-07-29T17:12:01.707+0000 7f2977bbc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T17:12:02.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:01 smithi129 bash[49373]: debug 2023-07-29T17:12:01.927+0000 7f2977bbc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T17:12:02.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:02 smithi129 bash[23526]: cluster 2023-07-29T17:12:01.915918+0000 mgr.y (mgr.15066) 43 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:12:02.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:02 smithi129 bash[49373]: debug 2023-07-29T17:12:02.331+0000 7f2977bbc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T17:12:02.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:02 smithi078 bash[23704]: cluster 2023-07-29T17:12:01.915918+0000 mgr.y (mgr.15066) 43 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:12:02.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:02 smithi078 bash[28259]: cluster 2023-07-29T17:12:01.915918+0000 mgr.y (mgr.15066) 43 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-29T17:12:03.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:02 smithi129 bash[49373]: debug 2023-07-29T17:12:02.867+0000 7f2977bbc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:12:03.305 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:12:03.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:03 smithi129 bash[23526]: audit 2023-07-29T17:12:03.273638+0000 mgr.y (mgr.15066) 44 : audit [DBG] from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:03.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:03 smithi129 bash[49373]: debug 2023-07-29T17:12:03.491+0000 7f2977bbc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T17:12:03.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:03 smithi129 bash[49373]: debug 2023-07-29T17:12:03.619+0000 7f2977bbc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T17:12:03.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:03 smithi078 bash[23704]: audit 2023-07-29T17:12:03.273638+0000 mgr.y (mgr.15066) 44 : audit [DBG] from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:03.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:03 smithi078 bash[28259]: audit 2023-07-29T17:12:03.273638+0000 mgr.y (mgr.15066) 44 : audit [DBG] from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.096 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:12:04.096 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (119s) 37s ago 8m 10.1M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (76s) 0s ago 7m 24.8M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 0s ago 5m - - 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (8s) 0s ago 12m 102M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (63s) 37s ago 14m 456M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (14m) 37s ago 14m 60.0M 2048M 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (12m) 0s ago 12m 57.1M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:12:04.097 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (12m) 37s ago 12m 48.9M 2048M 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2m) 37s ago 8m 8880k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (103s) 0s ago 8m 3483k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (12m) 37s ago 12m 55.0M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (11m) 37s ago 11m 52.6M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (11m) 37s ago 11m 47.3M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10m) 37s ago 10m 52.9M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:12:04.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (10m) 0s ago 10m 54.6M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (10m) 0s ago 10m 52.0M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (9m) 0s ago 9m 52.5M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (9m) 0s ago 9m 55.6M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 0s ago 8m - - 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (6m) 37s ago 6m 53.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (6m) 0s ago 6m 53.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (6m) 37s ago 6m 53.1M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:12:04.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (6m) 0s ago 6m 53.6M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:12:04.304 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.011+0000 7f2977bbc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T17:12:04.304 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.131+0000 7f2977bbc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T17:12:04.523 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:12:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:12:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-29T17:12:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:12:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:12:04.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:04.525 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:12:04.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2023-07-29T17:12:04.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:12:04.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:12:04.526 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:12:04.651 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.283+0000 7f2977bbc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T17:12:04.652 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.403+0000 7f2977bbc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T17:12:04.652 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.531+0000 7f2977bbc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T17:12:04.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:03.632242+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:03.639317+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:03.647800+0000 mgr.y (mgr.15066) 45 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: cluster 2023-07-29T17:12:03.916650+0000 mgr.y (mgr.15066) 46 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:04.892 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:12:04.893 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "2/23 daemons upgraded", 2023-07-29T17:12:04.894 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons", 2023-07-29T17:12:04.894 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:12:04.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:12:04.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.731+0000 7f2977bbc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T17:12:04.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[49373]: debug 2023-07-29T17:12:04.855+0000 7f2977bbc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T17:12:04.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:04.006781+0000 mgr.y (mgr.15066) 47 : audit [DBG] from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:04.385039+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:04.394083+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:04 smithi129 bash[23526]: audit 2023-07-29T17:12:04.506158+0000 mon.a (mon.0) 818 : audit [DBG] from='client.? 172.21.15.78:0/4287356686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:04.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:03.632242+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:03.639317+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:03.647800+0000 mgr.y (mgr.15066) 45 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: cluster 2023-07-29T17:12:03.916650+0000 mgr.y (mgr.15066) 46 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:04.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:04.006781+0000 mgr.y (mgr.15066) 47 : audit [DBG] from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:04.385039+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:04.394083+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[23704]: audit 2023-07-29T17:12:04.506158+0000 mon.a (mon.0) 818 : audit [DBG] from='client.? 172.21.15.78:0/4287356686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:03.632242+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:03.639317+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:03.647800+0000 mgr.y (mgr.15066) 45 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: cluster 2023-07-29T17:12:03.916650+0000 mgr.y (mgr.15066) 46 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:04.006781+0000 mgr.y (mgr.15066) 47 : audit [DBG] from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:04.385039+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:04.394083+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:04.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:04 smithi078 bash[28259]: audit 2023-07-29T17:12:04.506158+0000 mon.a (mon.0) 818 : audit [DBG] from='client.? 172.21.15.78:0/4287356686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:05.323 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-29T17:12:05.323 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-29T17:12:05.323 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:12:05.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:05 smithi129 bash[23526]: audit 2023-07-29T17:12:04.874635+0000 mgr.y (mgr.15066) 48 : audit [DBG] from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:05.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:05 smithi129 bash[23526]: audit 2023-07-29T17:12:05.305542+0000 mon.a (mon.0) 819 : audit [DBG] from='client.? 172.21.15.78:0/1466875185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:12:05.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:05 smithi078 bash[23704]: audit 2023-07-29T17:12:04.874635+0000 mgr.y (mgr.15066) 48 : audit [DBG] from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:05.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:05 smithi078 bash[23704]: audit 2023-07-29T17:12:05.305542+0000 mon.a (mon.0) 819 : audit [DBG] from='client.? 172.21.15.78:0/1466875185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:12:05.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:05 smithi078 bash[28259]: audit 2023-07-29T17:12:04.874635+0000 mgr.y (mgr.15066) 48 : audit [DBG] from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:05.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:05 smithi078 bash[28259]: audit 2023-07-29T17:12:05.305542+0000 mon.a (mon.0) 819 : audit [DBG] from='client.? 172.21.15.78:0/1466875185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:12:06.420 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[49373]: debug 2023-07-29T17:12:06.135+0000 7f2977bbc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T17:12:06.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: cluster 2023-07-29T17:12:05.918321+0000 mgr.y (mgr.15066) 49 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:06.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: cluster 2023-07-29T17:12:06.144101+0000 mon.a (mon.0) 820 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:12:06.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: cluster 2023-07-29T17:12:06.144312+0000 mon.a (mon.0) 821 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:12:06.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: audit 2023-07-29T17:12:06.149146+0000 mon.b (mon.1) 88 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:12:06.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: audit 2023-07-29T17:12:06.149875+0000 mon.b (mon.1) 89 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:12:06.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: audit 2023-07-29T17:12:06.151149+0000 mon.b (mon.1) 90 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:12:06.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:06 smithi129 bash[23526]: audit 2023-07-29T17:12:06.151827+0000 mon.b (mon.1) 91 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:12:06.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: cluster 2023-07-29T17:12:05.918321+0000 mgr.y (mgr.15066) 49 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: cluster 2023-07-29T17:12:06.144101+0000 mon.a (mon.0) 820 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: cluster 2023-07-29T17:12:06.144312+0000 mon.a (mon.0) 821 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: audit 2023-07-29T17:12:06.149146+0000 mon.b (mon.1) 88 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: audit 2023-07-29T17:12:06.149875+0000 mon.b (mon.1) 89 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: audit 2023-07-29T17:12:06.151149+0000 mon.b (mon.1) 90 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[23704]: audit 2023-07-29T17:12:06.151827+0000 mon.b (mon.1) 91 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:12:06.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: cluster 2023-07-29T17:12:05.918321+0000 mgr.y (mgr.15066) 49 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:06.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: cluster 2023-07-29T17:12:06.144101+0000 mon.a (mon.0) 820 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:12:06.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: cluster 2023-07-29T17:12:06.144312+0000 mon.a (mon.0) 821 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:12:06.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: audit 2023-07-29T17:12:06.149146+0000 mon.b (mon.1) 88 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:12:06.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: audit 2023-07-29T17:12:06.149875+0000 mon.b (mon.1) 89 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:12:06.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: audit 2023-07-29T17:12:06.151149+0000 mon.b (mon.1) 90 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:12:06.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:06 smithi078 bash[28259]: audit 2023-07-29T17:12:06.151827+0000 mon.b (mon.1) 91 : audit [DBG] from='mgr.? 172.21.15.129:0/387409011' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:12:07.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:07 smithi078 bash[23704]: cluster 2023-07-29T17:12:06.663058+0000 mon.a (mon.0) 822 : cluster [DBG] mgrmap e26: y(active, since 47s), standbys: x 2023-07-29T17:12:07.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:07 smithi078 bash[23704]: audit 2023-07-29T17:12:06.723049+0000 mon.b (mon.1) 92 : audit [DBG] from='client.? 172.21.15.129:0/3456265456' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:07.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:07 smithi078 bash[23704]: audit 2023-07-29T17:12:07.082380+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.129:0/582220644' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:12:07.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:07 smithi078 bash[28259]: cluster 2023-07-29T17:12:06.663058+0000 mon.a (mon.0) 822 : cluster [DBG] mgrmap e26: y(active, since 47s), standbys: x 2023-07-29T17:12:07.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:07 smithi078 bash[28259]: audit 2023-07-29T17:12:06.723049+0000 mon.b (mon.1) 92 : audit [DBG] from='client.? 172.21.15.129:0/3456265456' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:07.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:07 smithi078 bash[28259]: audit 2023-07-29T17:12:07.082380+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.129:0/582220644' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:12:08.000 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:07 smithi129 bash[23526]: cluster 2023-07-29T17:12:06.663058+0000 mon.a (mon.0) 822 : cluster [DBG] mgrmap e26: y(active, since 47s), standbys: x 2023-07-29T17:12:08.000 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:07 smithi129 bash[23526]: audit 2023-07-29T17:12:06.723049+0000 mon.b (mon.1) 92 : audit [DBG] from='client.? 172.21.15.129:0/3456265456' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:08.000 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:07 smithi129 bash[23526]: audit 2023-07-29T17:12:07.082380+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.129:0/582220644' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:12:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:08 smithi078 bash[23704]: cluster 2023-07-29T17:12:07.919145+0000 mgr.y (mgr.15066) 50 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:08.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:08 smithi078 bash[28259]: cluster 2023-07-29T17:12:07.919145+0000 mgr.y (mgr.15066) 50 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:09.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:08 smithi129 bash[23526]: cluster 2023-07-29T17:12:07.919145+0000 mgr.y (mgr.15066) 50 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:09.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:08 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2023-07-29T17:12:09.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:08 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:09.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:08 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:10.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.161Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:12:10.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.161Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:12:10.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.161Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:12:10.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.161Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:12:10.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.161Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:12:10.414 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.161Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.163Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.163Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.171Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.171Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=9 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.185Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=9 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.185Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=9 2023-07-29T17:12:10.415 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.185Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.186Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.186Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.187Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.188Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.188Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.190Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=9 2023-07-29T17:12:10.416 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.190Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=19.116227ms 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.193Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.193Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.193Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:12:10.417 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=info ts=2023-07-29T17:12:10.194Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:12:10.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[50578]: level=error ts=2023-07-29T17:12:10.194Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:12:11.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:10 smithi129 bash[23526]: cluster 2023-07-29T17:12:09.919949+0000 mgr.y (mgr.15066) 51 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:11.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:10 smithi078 bash[23704]: cluster 2023-07-29T17:12:09.919949+0000 mgr.y (mgr.15066) 51 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:11.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:10 smithi078 bash[28259]: cluster 2023-07-29T17:12:09.919949+0000 mgr.y (mgr.15066) 51 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:11.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:11 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:12:11.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:11 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:12:12.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:12 smithi129 bash[23526]: cluster 2023-07-29T17:12:11.921661+0000 mgr.y (mgr.15066) 52 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:12.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:12 smithi078 bash[23704]: cluster 2023-07-29T17:12:11.921661+0000 mgr.y (mgr.15066) 52 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:12.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:12 smithi078 bash[28259]: cluster 2023-07-29T17:12:11.921661+0000 mgr.y (mgr.15066) 52 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:15.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:14 smithi129 bash[23526]: cluster 2023-07-29T17:12:13.922468+0000 mgr.y (mgr.15066) 53 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:15.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:14 smithi078 bash[23704]: cluster 2023-07-29T17:12:13.922468+0000 mgr.y (mgr.15066) 53 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:15.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:14 smithi078 bash[28259]: cluster 2023-07-29T17:12:13.922468+0000 mgr.y (mgr.15066) 53 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:17.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: cluster 2023-07-29T17:12:15.924172+0000 mgr.y (mgr.15066) 54 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:17.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.252564+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.264504+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.266440+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.268121+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: cephadm 2023-07-29T17:12:16.269542+0000 mgr.y (mgr.15066) 55 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: cluster 2023-07-29T17:12:16.270737+0000 mgr.y (mgr.15066) 56 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: cluster 2023-07-29T17:12:16.271156+0000 mgr.y (mgr.15066) 57 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.280813+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.368583+0000 mon.a (mon.0) 828 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.372811+0000 mon.a (mon.0) 829 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: cephadm 2023-07-29T17:12:16.373930+0000 mgr.y (mgr.15066) 58 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.385060+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.386776+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.393812+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.395048+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-29T17:12:17.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.405025+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-07-29T17:12:17.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.407630+0000 mon.a (mon.0) 835 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:12:17.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: audit 2023-07-29T17:12:16.409240+0000 mon.a (mon.0) 836 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-07-29T17:12:17.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[23704]: cephadm 2023-07-29T17:12:16.410372+0000 mgr.y (mgr.15066) 59 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-07-29T17:12:17.657 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: cluster 2023-07-29T17:12:15.924172+0000 mgr.y (mgr.15066) 54 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:17.657 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.252564+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.264504+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.266440+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.268121+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: cephadm 2023-07-29T17:12:16.269542+0000 mgr.y (mgr.15066) 55 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: cluster 2023-07-29T17:12:16.270737+0000 mgr.y (mgr.15066) 56 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: cluster 2023-07-29T17:12:16.271156+0000 mgr.y (mgr.15066) 57 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.280813+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.368583+0000 mon.a (mon.0) 828 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.372811+0000 mon.a (mon.0) 829 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: cephadm 2023-07-29T17:12:16.373930+0000 mgr.y (mgr.15066) 58 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.385060+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.386776+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.393812+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.395048+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-29T17:12:17.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.405025+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-07-29T17:12:17.660 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.407630+0000 mon.a (mon.0) 835 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:12:17.660 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: audit 2023-07-29T17:12:16.409240+0000 mon.a (mon.0) 836 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-07-29T17:12:17.660 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:17 smithi078 bash[28259]: cephadm 2023-07-29T17:12:16.410372+0000 mgr.y (mgr.15066) 59 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-07-29T17:12:17.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: cluster 2023-07-29T17:12:15.924172+0000 mgr.y (mgr.15066) 54 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:17.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.252564+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.264504+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.266440+0000 mon.a (mon.0) 825 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:17.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.268121+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:12:17.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: cephadm 2023-07-29T17:12:16.269542+0000 mgr.y (mgr.15066) 55 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: cluster 2023-07-29T17:12:16.270737+0000 mgr.y (mgr.15066) 56 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: cluster 2023-07-29T17:12:16.271156+0000 mgr.y (mgr.15066) 57 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.280813+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.368583+0000 mon.a (mon.0) 828 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.372811+0000 mon.a (mon.0) 829 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: cephadm 2023-07-29T17:12:16.373930+0000 mgr.y (mgr.15066) 58 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-07-29T17:12:17.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.385060+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.386776+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.393812+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.395048+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.405025+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.407630+0000 mon.a (mon.0) 835 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: audit 2023-07-29T17:12:16.409240+0000 mon.a (mon.0) 836 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-07-29T17:12:17.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:17 smithi129 bash[23526]: cephadm 2023-07-29T17:12:16.410372+0000 mgr.y (mgr.15066) 59 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-07-29T17:12:18.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:18 smithi078 bash[23704]: cluster 2023-07-29T17:12:17.259779+0000 mon.a (mon.0) 837 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:12:18.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:18 smithi078 bash[28259]: cluster 2023-07-29T17:12:17.259779+0000 mon.a (mon.0) 837 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:12:18.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:18 smithi129 bash[23526]: cluster 2023-07-29T17:12:17.259779+0000 mon.a (mon.0) 837 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:12:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[23704]: cluster 2023-07-29T17:12:18.272009+0000 mgr.y (mgr.15066) 60 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-29T17:12:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[23704]: audit 2023-07-29T17:12:18.642744+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[23704]: audit 2023-07-29T17:12:18.644474+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:12:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[23704]: audit 2023-07-29T17:12:18.645902+0000 mon.a (mon.0) 840 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:12:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[23704]: audit 2023-07-29T17:12:18.647301+0000 mon.a (mon.0) 841 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:19.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:19 smithi129 bash[23526]: cluster 2023-07-29T17:12:18.272009+0000 mgr.y (mgr.15066) 60 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-29T17:12:19.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:19 smithi129 bash[23526]: audit 2023-07-29T17:12:18.642744+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:19.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:19 smithi129 bash[23526]: audit 2023-07-29T17:12:18.644474+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:12:19.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:19 smithi129 bash[23526]: audit 2023-07-29T17:12:18.645902+0000 mon.a (mon.0) 840 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:12:19.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:19 smithi129 bash[23526]: audit 2023-07-29T17:12:18.647301+0000 mon.a (mon.0) 841 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:19.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[28259]: cluster 2023-07-29T17:12:18.272009+0000 mgr.y (mgr.15066) 60 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-29T17:12:19.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[28259]: audit 2023-07-29T17:12:18.642744+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' 2023-07-29T17:12:19.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[28259]: audit 2023-07-29T17:12:18.644474+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:12:19.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[28259]: audit 2023-07-29T17:12:18.645902+0000 mon.a (mon.0) 840 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:12:19.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:19 smithi078 bash[28259]: audit 2023-07-29T17:12:18.647301+0000 mon.a (mon.0) 841 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:20.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:20 smithi078 bash[28259]: cephadm 2023-07-29T17:12:18.630942+0000 mgr.y (mgr.15066) 61 : cephadm [INF] Upgrade: Updating mon.a 2023-07-29T17:12:20.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:20 smithi078 bash[28259]: cephadm 2023-07-29T17:12:18.648913+0000 mgr.y (mgr.15066) 62 : cephadm [INF] Deploying daemon mon.a on smithi078 2023-07-29T17:12:20.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:20 smithi078 bash[23704]: cephadm 2023-07-29T17:12:18.630942+0000 mgr.y (mgr.15066) 61 : cephadm [INF] Upgrade: Updating mon.a 2023-07-29T17:12:20.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:20 smithi078 bash[23704]: cephadm 2023-07-29T17:12:18.648913+0000 mgr.y (mgr.15066) 62 : cephadm [INF] Deploying daemon mon.a on smithi078 2023-07-29T17:12:20.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:20 smithi129 bash[23526]: cephadm 2023-07-29T17:12:18.630942+0000 mgr.y (mgr.15066) 61 : cephadm [INF] Upgrade: Updating mon.a 2023-07-29T17:12:20.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:20 smithi129 bash[23526]: cephadm 2023-07-29T17:12:18.648913+0000 mgr.y (mgr.15066) 62 : cephadm [INF] Deploying daemon mon.a on smithi078 2023-07-29T17:12:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:21 smithi078 systemd[1]: Stopping Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:12:21.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:21 smithi129 bash[23526]: cluster 2023-07-29T17:12:20.272925+0000 mgr.y (mgr.15066) 63 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:21.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:21 smithi078 bash[28259]: cluster 2023-07-29T17:12:20.272925+0000 mgr.y (mgr.15066) 63 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:21.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:21 smithi078 bash[23704]: cluster 2023-07-29T17:12:20.272925+0000 mgr.y (mgr.15066) 63 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:21.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:21 smithi078 bash[23704]: debug 2023-07-29T17:12:21.279+0000 7f73a1f7a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T17:12:21.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:21 smithi078 bash[23704]: debug 2023-07-29T17:12:21.279+0000 7f73a1f7a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-29T17:12:22.170 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:21 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2023-07-29T17:12:22.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:21 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:22.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:21 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:22.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:21 smithi078 bash[58451]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mon.a 2023-07-29T17:12:22.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:22 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service: Succeeded. 2023-07-29T17:12:22.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:22 smithi078 systemd[1]: Stopped Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:22.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:22 smithi078 systemd[1]: Started Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:23.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.562+0000 7f594e224c80 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-29T17:12:23.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.562+0000 7f594e224c80 0 ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc), process ceph-mon, pid 7 2023-07-29T17:12:23.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.562+0000 7f594e224c80 0 pidfile_write: ignore empty --pid-file 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.566+0000 7f594e224c80 0 load: jerasure load: lrc 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: RocksDB version: 7.9.2 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Git sha 0 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Compile date 2023-07-28 16:56:26 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: DB SUMMARY 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: DB Session ID: YSV1R15GXMQ94DXTVW4S 2023-07-29T17:12:23.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: CURRENT file: CURRENT 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: IDENTITY file: IDENTITY 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 958 Bytes 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000023.sst 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000021.log size: 8904818 ; 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.error_if_exists: 0 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.create_if_missing: 0 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.paranoid_checks: 1 2023-07-29T17:12:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.env: 0x559188fd3720 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.fs: PosixFileSystem 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.info_log: 0x55918c0d9400 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-29T17:12:23.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.statistics: (nil) 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.use_fsync: 0 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_log_file_size: 0 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.allow_fallocate: 1 2023-07-29T17:12:23.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.use_direct_reads: 0 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.create_missing_column_families: 0 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.db_log_dir: 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.wal_dir: 2023-07-29T17:12:23.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.advise_random_on_open: 1 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-29T17:12:23.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.write_buffer_manager: 0x55918b366c80 2023-07-29T17:12:23.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T17:12:23.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T17:12:23.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T17:12:23.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.rate_limiter: (nil) 2023-07-29T17:12:23.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T17:12:23.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-29T17:12:23.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-29T17:12:23.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-29T17:12:23.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.unordered_write: 0 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.row_cache: None 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.wal_filter: None 2023-07-29T17:12:23.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.two_write_queues: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.manual_wal_flush: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.wal_compression: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.atomic_flush: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T17:12:23.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.log_readahead_size: 0 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.best_efforts_recovery: 0 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.allow_data_in_errors: 0 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.db_host_id: __hostname__ 2023-07-29T17:12:23.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_background_jobs: 2 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_background_compactions: -1 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_subcompactions: 1 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T17:12:23.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_total_wal_size: 0 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_open_files: -1 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bytes_per_sync: 0 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T17:12:23.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_background_flushes: -1 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Compression algorithms supported: 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kZSTD supported: 0 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kXpressCompression supported: 0 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-29T17:12:23.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kZlibCompression supported: 1 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kSnappyCompression supported: 1 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kLZ4Compression supported: 1 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: kBZip2Compression supported: 0 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2023-07-29T17:12:23.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.merge_operator: 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_filter: None 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_filter_factory: None 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.sst_partitioner_factory: None 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T17:12:23.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55918c0d9520) 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: cache_index_and_filter_blocks: 1 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: pin_top_level_index_and_filter: 1 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: index_type: 0 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: data_block_index_type: 0 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: index_shortening: 1 2023-07-29T17:12:23.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: data_block_hash_table_util_ratio: 0.750000 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: checksum: 4 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: no_block_cache: 0 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_cache: 0x55918b3e91f0 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_cache_name: BinnedLRUCache 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_cache_options: 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: capacity : 536870912 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: num_shard_bits : 4 2023-07-29T17:12:23.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: strict_capacity_limit : 0 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: high_pri_pool_ratio: 0.000 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_cache_compressed: (nil) 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: persistent_cache: (nil) 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_size: 4096 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_size_deviation: 10 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_restart_interval: 16 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: index_block_restart_interval: 1 2023-07-29T17:12:23.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: metadata_block_size: 4096 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: partition_filters: 0 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: use_delta_encoding: 1 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: filter_policy: bloomfilter 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: whole_key_filtering: 1 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: verify_compression: 0 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: read_amp_bytes_per_bit: 0 2023-07-29T17:12:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: format_version: 5 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: enable_index_compression: 1 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: block_align: 0 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: max_auto_readahead_size: 262144 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: prepopulate_block_cache: 0 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: initial_auto_readahead_size: 8192 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: num_file_reads_for_auto_readahead: 2 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-29T17:12:23.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression: NoCompression 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.num_levels: 7 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T17:12:23.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T17:12:23.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T17:12:23.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T17:12:23.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T17:12:23.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T17:12:23.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T17:12:23.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.level: 32767 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T17:12:23.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.enabled: false 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T17:12:23.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T17:12:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-29T17:12:23.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.arena_block_size: 1048576 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T17:12:23.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.table_properties_collectors: 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.inplace_update_support: 0 2023-07-29T17:12:23.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.bloom_locality: 0 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.max_successive_merges: 0 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T17:12:23.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.force_consistency_checks: 1 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.ttl: 2592000 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.enable_blob_files: false 2023-07-29T17:12:23.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.min_blob_size: 0 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.blob_file_size: 268435456 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-29T17:12:23.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.blob_file_starting_level: 0 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f5943a43700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 23.sst 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 25, last_sequence is 11332, log_number is 21,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 21 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0ce09d56-ef57-4b58-b21e-d1f559ceb06d 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650743573996, "job": 1, "event": "recovery_started", "wal_files": [21]} 2023-07-29T17:12:23.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.570+0000 7f594e224c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #21 mode 2 2023-07-29T17:12:23.699 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.438Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:12:23.699 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.439Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:12:23.700 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.439Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:12:23.700 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.439Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:12:23.700 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.439Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:12:23.700 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.439Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:12:23.700 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.441Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:12:23.700 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.441Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.446Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.446Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=10 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.458Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=10 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.458Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=10 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=10 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.461Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=10 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.462Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=10 2023-07-29T17:12:23.701 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.462Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=10 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.462Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=10 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.463Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=10 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.463Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=10 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.463Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=10 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.463Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=17.396396ms 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:12:23.702 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:12:23.703 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:12:23.704 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:12:23.704 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=info ts=2023-07-29T17:12:23.467Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:12:23.704 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:23 smithi129 bash[51108]: level=error ts=2023-07-29T17:12:23.468Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:12:23.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.602+0000 7f594e224c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650743604132, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 8651609, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11333, "largest_seqno": 13167, "table_properties": {"data_size": 8640312, "index_size": 8742, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 12258, "raw_average_key_size": 20, "raw_value_size": 8622762, "raw_average_value_size": 14639, "num_data_blocks": 384, "num_entries": 589, "num_filter_entries": 589, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1690650743, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0ce09d56-ef57-4b58-b21e-d1f559ceb06d", "db_session_id": "YSV1R15GXMQ94DXTVW4S", "orig_file_number": 26, "seqno_to_time_mapping": "N/A"}} 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.602+0000 7f594e224c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650743604373, "job": 1, "event": "recovery_finished"} 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.602+0000 7f594e224c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 28 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.602+0000 7f594e224c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.606+0000 7f594e224c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000021.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.606+0000 7f594e224c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55918b48a000 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.606+0000 7f594e224c80 4 rocksdb: DB pointer 0x55918b478000 2023-07-29T17:12:23.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.606+0000 7f593ca35700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.606+0000 7f593ca35700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: ** DB Stats ** 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T17:12:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T17:12:23.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T17:12:23.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T17:12:23.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: ** Compaction Stats [default] ** 2023-07-29T17:12:23.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: L0 1/0 8.25 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 518.3 0.02 0.00 1 0.016 0 0 0.0 0.0 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: L6 1/0 23.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Sum 2/0 31.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 518.3 0.02 0.00 1 0.016 0 0 0.0 0.0 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 518.3 0.02 0.00 1 0.016 0 0 0.0 0.0 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: ** Compaction Stats [default] ** 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T17:12:23.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 518.3 0.02 0.00 1 0.016 0 0 0.0 0.0 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Flush(GB): cumulative 0.008, interval 0.008 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T17:12:23.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Cumulative compaction: 0.01 GB write, 234.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Interval compaction: 0.01 GB write, 234.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Block cache BinnedLRUCache@0x55918b3e91f0#7 capacity: 512.00 MB usage: 10.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2e-05 secs_since: 0 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T17:12:23.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.606+0000 7f594e224c80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 1 mon.a@-1(???) e3 preinit fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 0 mon.a@-1(???).mds e1 new map 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 0 mon.a@-1(???).mds e1 print_map 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: e1 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: legacy client fscid: -1 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: No filesystems configured 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 0 mon.a@-1(???).osd e77 crush map has features 3314933000854323200, adjusting msgr requires 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 0 mon.a@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 0 mon.a@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 0 mon.a@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:12:23.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:12:23 smithi078 bash[58614]: debug 2023-07-29T17:12:23.610+0000 7f594e224c80 1 mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2023-07-29T17:12:24.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:24 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:12:24.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:24 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:12:25.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[49373]: ignoring --setuser ceph since I am not root 2023-07-29T17:12:25.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[49373]: ignoring --setgroup ceph since I am not root 2023-07-29T17:12:25.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[49373]: debug 2023-07-29T17:12:25.127+0000 7fd28d794700 1 -- 172.21.15.129:0/1246155727 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558080c9f1e0 con 0x558081a1e800 2023-07-29T17:12:25.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[49373]: debug 2023-07-29T17:12:25.287+0000 7fd2974c4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: audit 2023-07-29T17:12:21.291458+0000 mon.b (mon.1) 94 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: audit 2023-07-29T17:12:21.292044+0000 mon.b (mon.1) 95 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: audit 2023-07-29T17:12:21.292307+0000 mon.b (mon.1) 96 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: audit 2023-07-29T17:12:21.301303+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? 172.21.15.129:0/1362414489' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: audit 2023-07-29T17:12:21.672611+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.129:0/1296686860' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:22.275409+0000 mgr.y (mgr.15066) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:24.276211+0000 mgr.y (mgr.15066) 65 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.037530+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.040858+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.051451+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.052322+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.052373+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.053770+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e26: y(active, since 66s), standbys: x 2023-07-29T17:12:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.068975+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e27: y(active, since 66s), standbys: x 2023-07-29T17:12:25.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.069040+0000 mon.a (mon.0) 8 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:12:25.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.069063+0000 mon.a (mon.0) 9 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:12:25.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.069080+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi129 is in error state 2023-07-29T17:12:25.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: cluster 2023-07-29T17:12:25.069098+0000 mon.a (mon.0) 11 : cluster [WRN] daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:12:25.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:25 smithi129 bash[23526]: audit 2023-07-29T17:12:25.069214+0000 mon.a (mon.0) 12 : audit [INF] from='mgr.15066 ' entity='' 2023-07-29T17:12:25.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[55873]: ignoring --setuser ceph since I am not root 2023-07-29T17:12:25.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[55873]: ignoring --setgroup ceph since I am not root 2023-07-29T17:12:25.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[55873]: debug 2023-07-29T17:12:25.134+0000 7f903c8f1700 1 -- 172.21.15.78:0/4215239388 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5582d7aff1e0 con 0x5582d887ec00 2023-07-29T17:12:25.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[55873]: debug 2023-07-29T17:12:25.290+0000 7f9046621200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T17:12:25.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: audit 2023-07-29T17:12:21.291458+0000 mon.b (mon.1) 94 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:12:25.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: audit 2023-07-29T17:12:21.292044+0000 mon.b (mon.1) 95 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:12:25.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: audit 2023-07-29T17:12:21.292307+0000 mon.b (mon.1) 96 : audit [DBG] from='mgr.15066 172.21.15.78:0/1060280708' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:12:25.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: audit 2023-07-29T17:12:21.301303+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? 172.21.15.129:0/1362414489' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:25.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: audit 2023-07-29T17:12:21.672611+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.129:0/1296686860' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: access denied 2023-07-29T17:12:25.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:22.275409+0000 mgr.y (mgr.15066) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:24.276211+0000 mgr.y (mgr.15066) 65 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.037530+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.040858+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.051451+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.052322+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.052373+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-29T17:12:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.053770+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e26: y(active, since 66s), standbys: x 2023-07-29T17:12:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.068975+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e27: y(active, since 66s), standbys: x 2023-07-29T17:12:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.069040+0000 mon.a (mon.0) 8 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:12:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.069063+0000 mon.a (mon.0) 9 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:12:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.069080+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi129 is in error state 2023-07-29T17:12:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: cluster 2023-07-29T17:12:25.069098+0000 mon.a (mon.0) 11 : cluster [WRN] daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:12:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:25 smithi078 bash[28259]: audit 2023-07-29T17:12:25.069214+0000 mon.a (mon.0) 12 : audit [INF] from='mgr.15066 ' entity='' 2023-07-29T17:12:26.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:26 smithi078 bash[55873]: debug 2023-07-29T17:12:26.166+0000 7f9046621200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T17:12:26.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:26 smithi078 bash[55873]: debug 2023-07-29T17:12:26.262+0000 7f9046621200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T17:12:26.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:26 smithi078 bash[55873]: debug 2023-07-29T17:12:26.362+0000 7f9046621200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T17:12:26.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:26 smithi129 bash[49373]: debug 2023-07-29T17:12:26.199+0000 7fd2974c4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T17:12:26.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:26 smithi129 bash[49373]: debug 2023-07-29T17:12:26.299+0000 7fd2974c4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T17:12:26.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:26 smithi129 bash[49373]: debug 2023-07-29T17:12:26.399+0000 7fd2974c4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T17:12:26.939 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:26 smithi078 bash[55873]: debug 2023-07-29T17:12:26.618+0000 7f9046621200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:12:27.026 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:26 smithi129 bash[49373]: debug 2023-07-29T17:12:26.659+0000 7fd2974c4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:12:27.358 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:26 smithi078 bash[55873]: debug 2023-07-29T17:12:26.966+0000 7f9046621200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T17:12:27.358 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:27 smithi078 bash[55873]: debug 2023-07-29T17:12:27.126+0000 7f9046621200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T17:12:27.358 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:27 smithi078 bash[28259]: cluster 2023-07-29T17:12:26.076022+0000 mon.a (mon.0) 13 : cluster [DBG] mgrmap e28: y(active, since 67s), standbys: x 2023-07-29T17:12:27.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:27 smithi129 bash[23526]: cluster 2023-07-29T17:12:26.076022+0000 mon.a (mon.0) 13 : cluster [DBG] mgrmap e28: y(active, since 67s), standbys: x 2023-07-29T17:12:27.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:27 smithi129 bash[49373]: debug 2023-07-29T17:12:27.003+0000 7fd2974c4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T17:12:27.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:27 smithi129 bash[49373]: debug 2023-07-29T17:12:27.179+0000 7fd2974c4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T17:12:27.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:27 smithi078 bash[55873]: debug 2023-07-29T17:12:27.338+0000 7f9046621200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T17:12:27.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:27 smithi129 bash[49373]: debug 2023-07-29T17:12:27.407+0000 7fd2974c4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T17:12:28.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:28 smithi078 bash[55873]: debug 2023-07-29T17:12:28.318+0000 7f9046621200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T17:12:28.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:28 smithi078 bash[55873]: debug 2023-07-29T17:12:28.430+0000 7f9046621200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T17:12:28.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:28 smithi078 bash[55873]: debug 2023-07-29T17:12:28.546+0000 7f9046621200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T17:12:28.922 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:28 smithi129 bash[49373]: debug 2023-07-29T17:12:28.431+0000 7fd2974c4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T17:12:28.922 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:28 smithi129 bash[49373]: debug 2023-07-29T17:12:28.543+0000 7fd2974c4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T17:12:28.922 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:28 smithi129 bash[49373]: debug 2023-07-29T17:12:28.667+0000 7fd2974c4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T17:12:29.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:29 smithi078 bash[55873]: debug 2023-07-29T17:12:29.010+0000 7f9046621200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T17:12:29.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:29 smithi078 bash[55873]: debug 2023-07-29T17:12:29.230+0000 7f9046621200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T17:12:29.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:29 smithi129 bash[49373]: debug 2023-07-29T17:12:29.163+0000 7fd2974c4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T17:12:29.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:29 smithi129 bash[49373]: debug 2023-07-29T17:12:29.395+0000 7fd2974c4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T17:12:29.939 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:29 smithi078 bash[55873]: debug 2023-07-29T17:12:29.598+0000 7f9046621200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T17:12:30.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:29 smithi129 bash[49373]: debug 2023-07-29T17:12:29.791+0000 7fd2974c4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T17:12:30.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:30 smithi078 bash[55873]: debug 2023-07-29T17:12:30.086+0000 7f9046621200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:12:30.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:30 smithi129 bash[49373]: debug 2023-07-29T17:12:30.283+0000 7fd2974c4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T17:12:30.939 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:30 smithi078 bash[55873]: debug 2023-07-29T17:12:30.630+0000 7f9046621200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T17:12:30.939 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:30 smithi078 bash[55873]: debug 2023-07-29T17:12:30.738+0000 7f9046621200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T17:12:31.170 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:30 smithi129 bash[49373]: debug 2023-07-29T17:12:30.827+0000 7fd2974c4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T17:12:31.171 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:30 smithi129 bash[49373]: debug 2023-07-29T17:12:30.939+0000 7fd2974c4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T17:12:31.355 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.082+0000 7f9046621200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T17:12:31.355 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.194+0000 7f9046621200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T17:12:31.562 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:31 smithi129 bash[49373]: debug 2023-07-29T17:12:31.283+0000 7fd2974c4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T17:12:31.562 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:31 smithi129 bash[49373]: debug 2023-07-29T17:12:31.399+0000 7fd2974c4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T17:12:31.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.334+0000 7f9046621200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T17:12:31.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.450+0000 7f9046621200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T17:12:31.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.574+0000 7f9046621200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T17:12:31.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:31 smithi129 bash[49373]: debug 2023-07-29T17:12:31.539+0000 7fd2974c4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T17:12:31.921 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:31 smithi129 bash[49373]: debug 2023-07-29T17:12:31.659+0000 7fd2974c4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T17:12:31.921 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:31 smithi129 bash[49373]: debug 2023-07-29T17:12:31.779+0000 7fd2974c4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T17:12:32.189 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.758+0000 7f9046621200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T17:12:32.189 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:31 smithi078 bash[55873]: debug 2023-07-29T17:12:31.882+0000 7f9046621200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T17:12:32.420 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:31 smithi129 bash[49373]: debug 2023-07-29T17:12:31.967+0000 7fd2974c4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T17:12:32.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:32 smithi129 bash[49373]: debug 2023-07-29T17:12:32.091+0000 7fd2974c4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T17:12:33.400 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[55873]: debug 2023-07-29T17:12:33.094+0000 7f9046621200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T17:12:33.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: debug 2023-07-29T17:12:33.126+0000 7f82efa4e700 -1 mon.c@2(peon).osd e78 update_from_paxos full map CRC mismatch, resetting to canonical 2023-07-29T17:12:33.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.119267+0000 mon.a (mon.0) 14 : cluster [INF] Active manager daemon y restarted 2023-07-29T17:12:33.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.119990+0000 mon.a (mon.0) 15 : cluster [INF] Activating manager daemon y 2023-07-29T17:12:33.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.132054+0000 mon.a (mon.0) 16 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:12:33.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.132684+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e29: y(active, starting, since 0.0128629s), standbys: x 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.150826+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.151107+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.151347+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.153565+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.153921+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.154157+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:12:33.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.154444+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.154715+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.155069+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.155408+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.155717+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.156010+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.156279+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.156614+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:12:33.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.156832+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:12:33.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: audit 2023-07-29T17:12:33.157353+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:12:33.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:33 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.165504+0000 mon.a (mon.0) 34 : cluster [INF] Manager daemon y is now available 2023-07-29T17:12:33.408 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[49373]: debug 2023-07-29T17:12:33.331+0000 7fd2974c4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T17:12:33.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: debug 2023-07-29T17:12:33.127+0000 7f1dfc2ba700 -1 mon.b@1(peon).osd e78 update_from_paxos full map CRC mismatch, resetting to canonical 2023-07-29T17:12:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.119267+0000 mon.a (mon.0) 14 : cluster [INF] Active manager daemon y restarted 2023-07-29T17:12:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.119990+0000 mon.a (mon.0) 15 : cluster [INF] Activating manager daemon y 2023-07-29T17:12:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.132054+0000 mon.a (mon.0) 16 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:12:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.132684+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e29: y(active, starting, since 0.0128629s), standbys: x 2023-07-29T17:12:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.150826+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:12:33.409 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.151107+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.151347+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.153565+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.153921+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.154157+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.154444+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.154715+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.155069+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:12:33.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.155408+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.155717+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.156010+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.156279+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.156614+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.156832+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: audit 2023-07-29T17:12:33.157353+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T17:12:33.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:33 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.165504+0000 mon.a (mon.0) 34 : cluster [INF] Manager daemon y is now available 2023-07-29T17:12:34.439 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[55873]: debug 2023-07-29T17:12:34.146+0000 7f900e65c700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-07-29T17:12:34.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.138359+0000 osd.5 (osd.5) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.139239+0000 osd.1 (osd.1) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.207644+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.225936+0000 osd.7 (osd.7) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.258815+0000 mon.a (mon.0) 36 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.290688+0000 mon.a (mon.0) 37 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.352681+0000 mon.a (mon.0) 38 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:12:34.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.352894+0000 mon.a (mon.0) 39 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.356200+0000 mon.b (mon.1) 99 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.360371+0000 mon.b (mon.1) 100 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.362026+0000 mon.b (mon.1) 101 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: audit 2023-07-29T17:12:33.362719+0000 mon.b (mon.1) 102 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cephadm 2023-07-29T17:12:33.691763+0000 mgr.y (mgr.25169) 1 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Bus STARTING 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cephadm 2023-07-29T17:12:33.818280+0000 mgr.y (mgr.25169) 2 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cephadm 2023-07-29T17:12:33.920427+0000 mgr.y (mgr.25169) 3 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:12:34.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cephadm 2023-07-29T17:12:33.921157+0000 mgr.y (mgr.25169) 4 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Bus STARTED 2023-07-29T17:12:34.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:34 smithi078 bash[28259]: cluster 2023-07-29T17:12:34.140406+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e30: y(active, since 1.02058s), standbys: x 2023-07-29T17:12:34.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.138359+0000 osd.5 (osd.5) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:34.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.139239+0000 osd.1 (osd.1) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.207644+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.225936+0000 osd.7 (osd.7) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.258815+0000 mon.a (mon.0) 36 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.290688+0000 mon.a (mon.0) 37 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.352681+0000 mon.a (mon.0) 38 : cluster [DBG] Standby manager daemon x restarted 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.352894+0000 mon.a (mon.0) 39 : cluster [DBG] Standby manager daemon x started 2023-07-29T17:12:34.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.356200+0000 mon.b (mon.1) 99 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.360371+0000 mon.b (mon.1) 100 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.362026+0000 mon.b (mon.1) 101 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: audit 2023-07-29T17:12:33.362719+0000 mon.b (mon.1) 102 : audit [DBG] from='mgr.? 172.21.15.129:0/4022568300' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cephadm 2023-07-29T17:12:33.691763+0000 mgr.y (mgr.25169) 1 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Bus STARTING 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cephadm 2023-07-29T17:12:33.818280+0000 mgr.y (mgr.25169) 2 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cephadm 2023-07-29T17:12:33.920427+0000 mgr.y (mgr.25169) 3 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cephadm 2023-07-29T17:12:33.921157+0000 mgr.y (mgr.25169) 4 : cephadm [INF] [29/Jul/2023:17:12:33] ENGINE Bus STARTED 2023-07-29T17:12:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:34 smithi129 bash[23526]: cluster 2023-07-29T17:12:34.140406+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e30: y(active, since 1.02058s), standbys: x 2023-07-29T17:12:35.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:34 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2023-07-29T17:12:35.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:34 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:35.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:34 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:35.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:35 smithi129 bash[23526]: cluster 2023-07-29T17:12:33.851254+0000 osd.2 (osd.2) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:35.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:35 smithi129 bash[23526]: cluster 2023-07-29T17:12:34.155361+0000 mgr.y (mgr.25169) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:35.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:35 smithi129 bash[23526]: cluster 2023-07-29T17:12:35.132649+0000 mon.a (mon.0) 41 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-29T17:12:35.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:35 smithi129 bash[23526]: cluster 2023-07-29T17:12:35.132713+0000 mon.a (mon.0) 42 : cluster [INF] Cluster is now healthy 2023-07-29T17:12:35.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:35 smithi078 bash[28259]: cluster 2023-07-29T17:12:33.851254+0000 osd.2 (osd.2) 3 : cluster [WRN] failed to encode map e78 with expected crc 2023-07-29T17:12:35.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:35 smithi078 bash[28259]: cluster 2023-07-29T17:12:34.155361+0000 mgr.y (mgr.25169) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:35.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:35 smithi078 bash[28259]: cluster 2023-07-29T17:12:35.132649+0000 mon.a (mon.0) 41 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-29T17:12:35.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:35 smithi078 bash[28259]: cluster 2023-07-29T17:12:35.132713+0000 mon.a (mon.0) 42 : cluster [INF] Cluster is now healthy 2023-07-29T17:12:35.711 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:12:36.456 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:12:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2m) 69s ago 8m 10.1M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:12:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (109s) 32s ago 8m 24.8M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:12:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 32s ago 6m - - 2023-07-29T17:12:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (40s) 32s ago 13m 102M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:12:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (96s) 69s ago 14m 456M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:12:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (14m) 69s ago 14m 60.0M 2048M 16.2.0 24ecd6d5f14c 892d448c0e05 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (13m) 32s ago 13m 57.1M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (13m) 69s ago 13m 48.9M 2048M 16.2.0 24ecd6d5f14c 2c90451fc7d2 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2m) 69s ago 8m 8880k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (2m) 32s ago 8m 3483k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (12m) 69s ago 12m 55.0M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (12m) 69s ago 12m 52.6M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:12:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (11m) 69s ago 11m 47.3M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (11m) 69s ago 11m 52.9M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (11m) 32s ago 11m 54.6M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (10m) 32s ago 10m 52.0M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (10m) 32s ago 10m 52.5M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (9m) 32s ago 9m 55.6M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 32s ago 9m - - 2023-07-29T17:12:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (6m) 69s ago 6m 53.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:12:36.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (6m) 32s ago 6m 53.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:12:36.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (6m) 69s ago 6m 53.1M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:12:36.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (6m) 32s ago 6m 53.6M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:12:36.897 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:12:36.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": {}, 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:12:36.899 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:12:36.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 14, 2023-07-29T17:12:36.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:12:36.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:12:36.900 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:12:36.917 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.836Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:12:36.917 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.836Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:12:36.917 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.836Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:12:36.917 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.836Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.836Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.836Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.838Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.838Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.841Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.841Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=11 2023-07-29T17:12:36.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.859Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.859Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.862Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=11 2023-07-29T17:12:36.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.863Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=11 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.863Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=11 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.863Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=11 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.863Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=22.025971ms 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.865Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.865Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.865Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:12:36.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:12:36.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:12:36.922 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=info ts=2023-07-29T17:12:36.866Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:12:36.922 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:36 smithi129 bash[51440]: level=error ts=2023-07-29T17:12:36.866Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:12:37.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:37 smithi129 bash[23526]: cluster 2023-07-29T17:12:35.155397+0000 mgr.y (mgr.25169) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:37.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:37 smithi129 bash[23526]: audit 2023-07-29T17:12:35.677721+0000 mgr.y (mgr.25169) 7 : audit [DBG] from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:37.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:37 smithi129 bash[23526]: audit 2023-07-29T17:12:36.055894+0000 mgr.y (mgr.25169) 8 : audit [DBG] from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:37.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:37 smithi129 bash[23526]: cluster 2023-07-29T17:12:36.151205+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e31: y(active, since 3s), standbys: x 2023-07-29T17:12:37.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:37 smithi129 bash[23526]: audit 2023-07-29T17:12:36.879152+0000 mon.c (mon.2) 32 : audit [DBG] from='client.34155 172.21.15.78:0/1986609401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:37.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:12:37.289 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:12:37.289 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:12:37.289 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:12:37.289 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:12:37.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2023-07-29T17:12:37.290 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:12:37.290 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "2/23 daemons upgraded", 2023-07-29T17:12:37.290 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "", 2023-07-29T17:12:37.290 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:12:37.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:12:37.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:37 smithi078 bash[28259]: cluster 2023-07-29T17:12:35.155397+0000 mgr.y (mgr.25169) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:37.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:37 smithi078 bash[28259]: audit 2023-07-29T17:12:35.677721+0000 mgr.y (mgr.25169) 7 : audit [DBG] from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:37.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:37 smithi078 bash[28259]: audit 2023-07-29T17:12:36.055894+0000 mgr.y (mgr.25169) 8 : audit [DBG] from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:37.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:37 smithi078 bash[28259]: cluster 2023-07-29T17:12:36.151205+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e31: y(active, since 3s), standbys: x 2023-07-29T17:12:37.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:37 smithi078 bash[28259]: audit 2023-07-29T17:12:36.879152+0000 mon.c (mon.2) 32 : audit [DBG] from='client.34155 172.21.15.78:0/1986609401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:37.723 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:12:37.930 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:37 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:12:37.930 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:37 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:12:38.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:38 smithi129 bash[23526]: audit 2023-07-29T17:12:36.416460+0000 mgr.y (mgr.25169) 9 : audit [DBG] from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:38.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:38 smithi129 bash[23526]: audit 2023-07-29T17:12:37.465597+0000 mon.b (mon.1) 103 : audit [DBG] from='client.? 172.21.15.129:0/4218879352' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:38.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:38 smithi129 bash[23526]: audit 2023-07-29T17:12:37.705334+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? 172.21.15.78:0/1197588760' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:12:38.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:38 smithi129 bash[23526]: audit 2023-07-29T17:12:37.843909+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.129:0/158598556' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:12:38.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:38 smithi078 bash[28259]: audit 2023-07-29T17:12:36.416460+0000 mgr.y (mgr.25169) 9 : audit [DBG] from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:38.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:38 smithi078 bash[28259]: audit 2023-07-29T17:12:37.465597+0000 mon.b (mon.1) 103 : audit [DBG] from='client.? 172.21.15.129:0/4218879352' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:38.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:38 smithi078 bash[28259]: audit 2023-07-29T17:12:37.705334+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? 172.21.15.78:0/1197588760' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:12:38.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:38 smithi078 bash[28259]: audit 2023-07-29T17:12:37.843909+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.129:0/158598556' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:12:39.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:39 smithi129 bash[23526]: cluster 2023-07-29T17:12:37.156085+0000 mgr.y (mgr.25169) 10 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:39.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:39 smithi129 bash[23526]: audit 2023-07-29T17:12:37.270148+0000 mgr.y (mgr.25169) 11 : audit [DBG] from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:39.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:39 smithi078 bash[28259]: cluster 2023-07-29T17:12:37.156085+0000 mgr.y (mgr.25169) 10 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:39.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:39 smithi078 bash[28259]: audit 2023-07-29T17:12:37.270148+0000 mgr.y (mgr.25169) 11 : audit [DBG] from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:12:40.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:40 smithi078 bash[28259]: cluster 2023-07-29T17:12:39.156986+0000 mgr.y (mgr.25169) 12 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:12:40.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:40 smithi129 bash[23526]: cluster 2023-07-29T17:12:39.156986+0000 mgr.y (mgr.25169) 12 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:12:41.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:41 smithi129 bash[23526]: audit 2023-07-29T17:12:40.182021+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:41 smithi129 bash[23526]: audit 2023-07-29T17:12:40.192107+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:41 smithi129 bash[23526]: audit 2023-07-29T17:12:40.893782+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:41 smithi129 bash[23526]: audit 2023-07-29T17:12:40.903310+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:41 smithi129 bash[23526]: audit 2023-07-29T17:12:40.990950+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:41 smithi129 bash[23526]: audit 2023-07-29T17:12:41.002809+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:41 smithi078 bash[28259]: audit 2023-07-29T17:12:40.182021+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:41 smithi078 bash[28259]: audit 2023-07-29T17:12:40.192107+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:41 smithi078 bash[28259]: audit 2023-07-29T17:12:40.893782+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:41 smithi078 bash[28259]: audit 2023-07-29T17:12:40.903310+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:41 smithi078 bash[28259]: audit 2023-07-29T17:12:40.990950+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:41.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:41 smithi078 bash[28259]: audit 2023-07-29T17:12:41.002809+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:42 smithi129 bash[23526]: cluster 2023-07-29T17:12:41.157681+0000 mgr.y (mgr.25169) 13 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2023-07-29T17:12:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:42 smithi129 bash[23526]: audit 2023-07-29T17:12:41.701420+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:43.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:42 smithi129 bash[23526]: audit 2023-07-29T17:12:41.712500+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:43.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:42 smithi078 bash[28259]: cluster 2023-07-29T17:12:41.157681+0000 mgr.y (mgr.25169) 13 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2023-07-29T17:12:43.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:42 smithi078 bash[28259]: audit 2023-07-29T17:12:41.701420+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:43.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:42 smithi078 bash[28259]: audit 2023-07-29T17:12:41.712500+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:45.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:44 smithi129 bash[23526]: cluster 2023-07-29T17:12:43.158401+0000 mgr.y (mgr.25169) 14 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2023-07-29T17:12:45.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:44 smithi078 bash[28259]: cluster 2023-07-29T17:12:43.158401+0000 mgr.y (mgr.25169) 14 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2023-07-29T17:12:46.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:46 smithi078 bash[28259]: cluster 2023-07-29T17:12:45.160084+0000 mgr.y (mgr.25169) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s 2023-07-29T17:12:47.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:46 smithi129 bash[23526]: cluster 2023-07-29T17:12:45.160084+0000 mgr.y (mgr.25169) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s 2023-07-29T17:12:48.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:47 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 5. 2023-07-29T17:12:48.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:47 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:48.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:47 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:12:49.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:48 smithi078 bash[28259]: cluster 2023-07-29T17:12:47.160880+0000 mgr.y (mgr.25169) 16 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:49.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:48 smithi129 bash[23526]: cluster 2023-07-29T17:12:47.160880+0000 mgr.y (mgr.25169) 16 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:50.740 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.549Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-29T17:12:50.740 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.549Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-29T17:12:50.740 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.549Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-29T17:12:50.741 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.549Z caller=main.go:339 host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:12:50.741 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.549Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:12:50.741 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.549Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:12:50.741 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.553Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-29T17:12:50.741 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.553Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-29T17:12:50.742 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.558Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-29T17:12:50.742 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.558Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=12 2023-07-29T17:12:50.742 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.573Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=12 2023-07-29T17:12:50.742 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.574Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=12 2023-07-29T17:12:50.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.574Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=12 2023-07-29T17:12:50.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.574Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=12 2023-07-29T17:12:50.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.574Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=12 2023-07-29T17:12:50.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.574Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=12 2023-07-29T17:12:50.744 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.575Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=12 2023-07-29T17:12:50.744 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.575Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=12 2023-07-29T17:12:50.744 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.575Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=12 2023-07-29T17:12:50.745 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.576Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=12 2023-07-29T17:12:50.745 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.576Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=12 2023-07-29T17:12:50.745 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.577Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=12 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.577Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=18.751793ms 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:695 msg="TSDB started" 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-29T17:12:50.746 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-29T17:12:50.747 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-29T17:12:50.747 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-29T17:12:50.747 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:12:50.747 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.581Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-29T17:12:50.747 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.582Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-29T17:12:50.747 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.582Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-29T17:12:50.748 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=info ts=2023-07-29T17:12:50.582Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-29T17:12:50.748 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[52517]: level=error ts=2023-07-29T17:12:50.582Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-29T17:12:50.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:50 smithi129 bash[23526]: cluster 2023-07-29T17:12:49.162452+0000 mgr.y (mgr.25169) 17 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:50.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:50 smithi078 bash[28259]: cluster 2023-07-29T17:12:49.162452+0000 mgr.y (mgr.25169) 17 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:52.191 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:51 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-29T17:12:52.191 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:12:51 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:12:52.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:52 smithi129 bash[23526]: cluster 2023-07-29T17:12:51.163203+0000 mgr.y (mgr.25169) 18 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:52.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:52 smithi129 bash[23526]: audit 2023-07-29T17:12:51.171698+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:52.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:52 smithi129 bash[23526]: audit 2023-07-29T17:12:51.181721+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:52.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:52 smithi078 bash[28259]: cluster 2023-07-29T17:12:51.163203+0000 mgr.y (mgr.25169) 18 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:52.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:52 smithi078 bash[28259]: audit 2023-07-29T17:12:51.171698+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:52.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:52 smithi078 bash[28259]: audit 2023-07-29T17:12:51.181721+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:54.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:54 smithi129 bash[23526]: cluster 2023-07-29T17:12:53.163956+0000 mgr.y (mgr.25169) 19 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:54.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:54 smithi129 bash[23526]: audit 2023-07-29T17:12:53.373130+0000 mon.b (mon.1) 105 : audit [DBG] from='client.? 172.21.15.129:0/720231089' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:54.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:54 smithi129 bash[23526]: audit 2023-07-29T17:12:53.744700+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.129:0/2122394065' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:12:54.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:54 smithi078 bash[28259]: cluster 2023-07-29T17:12:53.163956+0000 mgr.y (mgr.25169) 19 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:12:54.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:54 smithi078 bash[28259]: audit 2023-07-29T17:12:53.373130+0000 mon.b (mon.1) 105 : audit [DBG] from='client.? 172.21.15.129:0/720231089' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:12:54.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:54 smithi078 bash[28259]: audit 2023-07-29T17:12:53.744700+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.129:0/2122394065' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:12:56.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.125639+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:56.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.135001+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.137619+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.139021+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: cephadm 2023-07-29T17:12:55.140241+0000 mgr.y (mgr.25169) 20 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: cluster 2023-07-29T17:12:55.141962+0000 mgr.y (mgr.25169) 21 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.147652+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.234039+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:56.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.238749+0000 mon.a (mon.0) 60 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:56.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.242123+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:12:56.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:56 smithi129 bash[23526]: audit 2023-07-29T17:12:55.244043+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-29T17:12:56.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.125639+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:56.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.135001+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:56.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.137619+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:56.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.139021+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:12:56.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: cephadm 2023-07-29T17:12:55.140241+0000 mgr.y (mgr.25169) 20 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:12:56.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: cluster 2023-07-29T17:12:55.141962+0000 mgr.y (mgr.25169) 21 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:56.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.147652+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:56.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.234039+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:12:56.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.238749+0000 mon.a (mon.0) 60 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:12:56.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.242123+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:12:56.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:56 smithi078 bash[28259]: audit 2023-07-29T17:12:55.244043+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-29T17:12:57.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:57 smithi078 bash[28259]: cephadm 2023-07-29T17:12:55.245469+0000 mgr.y (mgr.25169) 22 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2023-07-29T17:12:57.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:57 smithi078 bash[28259]: cluster 2023-07-29T17:12:56.131866+0000 mon.a (mon.0) 63 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:12:57.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:57 smithi129 bash[23526]: cephadm 2023-07-29T17:12:55.245469+0000 mgr.y (mgr.25169) 22 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2023-07-29T17:12:57.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:57 smithi129 bash[23526]: cluster 2023-07-29T17:12:56.131866+0000 mon.a (mon.0) 63 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:12:58.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:58 smithi129 bash[23526]: cluster 2023-07-29T17:12:57.142651+0000 mgr.y (mgr.25169) 23 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:58.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:58 smithi129 bash[23526]: audit 2023-07-29T17:12:57.524116+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:58.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:58 smithi129 bash[23526]: audit 2023-07-29T17:12:57.525945+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:12:58.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:58 smithi129 bash[23526]: audit 2023-07-29T17:12:57.527053+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:12:58.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:58 smithi129 bash[23526]: audit 2023-07-29T17:12:57.528224+0000 mon.a (mon.0) 67 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:58.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:58 smithi078 bash[28259]: cluster 2023-07-29T17:12:57.142651+0000 mgr.y (mgr.25169) 23 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:12:58.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:58 smithi078 bash[28259]: audit 2023-07-29T17:12:57.524116+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:12:58.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:58 smithi078 bash[28259]: audit 2023-07-29T17:12:57.525945+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:12:58.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:58 smithi078 bash[28259]: audit 2023-07-29T17:12:57.527053+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:12:58.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:58 smithi078 bash[28259]: audit 2023-07-29T17:12:57.528224+0000 mon.a (mon.0) 67 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:12:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:59 smithi078 bash[28259]: cephadm 2023-07-29T17:12:57.514548+0000 mgr.y (mgr.25169) 24 : cephadm [INF] Upgrade: Updating mon.c 2023-07-29T17:12:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:12:59 smithi078 bash[28259]: cephadm 2023-07-29T17:12:57.529886+0000 mgr.y (mgr.25169) 25 : cephadm [INF] Deploying daemon mon.c on smithi078 2023-07-29T17:12:59.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:59 smithi129 bash[23526]: cephadm 2023-07-29T17:12:57.514548+0000 mgr.y (mgr.25169) 24 : cephadm [INF] Upgrade: Updating mon.c 2023-07-29T17:12:59.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:12:59 smithi129 bash[23526]: cephadm 2023-07-29T17:12:57.529886+0000 mgr.y (mgr.25169) 25 : cephadm [INF] Deploying daemon mon.c on smithi078 2023-07-29T17:13:00.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:00 smithi078 systemd[1]: Stopping Ceph mon.c for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:13:00.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:00 smithi129 bash[23526]: cluster 2023-07-29T17:12:59.143567+0000 mgr.y (mgr.25169) 26 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:00.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:00 smithi078 bash[28259]: cluster 2023-07-29T17:12:59.143567+0000 mgr.y (mgr.25169) 26 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:00.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:00 smithi078 bash[28259]: debug 2023-07-29T17:13:00.162+0000 7f82f625b700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T17:13:00.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:00 smithi078 bash[28259]: debug 2023-07-29T17:13:00.162+0000 7f82f625b700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2023-07-29T17:13:01.348 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:00 smithi078 bash[60196]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mon.c 2023-07-29T17:13:01.348 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:01 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.c.service: Succeeded. 2023-07-29T17:13:01.348 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:01 smithi078 systemd[1]: Stopped Ceph mon.c for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:13:01.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:01 smithi078 systemd[1]: Started Ceph mon.c for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:13:02.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:13:02 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 6. 2023-07-29T17:13:02.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:13:02 smithi129 systemd[1]: Stopped Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:13:02.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:13:02 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Start request repeated too quickly. 2023-07-29T17:13:02.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:13:02 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-29T17:13:02.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:13:02 smithi129 systemd[1]: Failed to start Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:13:02.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.518+0000 7f0cd1abfc80 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-29T17:13:02.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.518+0000 7f0cd1abfc80 0 ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc), process ceph-mon, pid 7 2023-07-29T17:13:02.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.522+0000 7f0cd1abfc80 0 pidfile_write: ignore empty --pid-file 2023-07-29T17:13:02.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 0 load: jerasure load: lrc 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: RocksDB version: 7.9.2 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Git sha 0 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Compile date 2023-07-28 16:56:26 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: DB SUMMARY 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: DB Session ID: DWYMMB3GOQ9QTT50GA4D 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: CURRENT file: CURRENT 2023-07-29T17:13:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: IDENTITY file: IDENTITY 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 882 Bytes 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000020.sst 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000018.log size: 18138620 ; 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.error_if_exists: 0 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.create_if_missing: 0 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.paranoid_checks: 1 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-07-29T17:13:02.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.env: 0x55c69bddb720 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.fs: PosixFileSystem 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.info_log: 0x55c69dd393e0 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.statistics: (nil) 2023-07-29T17:13:02.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.use_fsync: 0 2023-07-29T17:13:02.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_log_file_size: 0 2023-07-29T17:13:02.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T17:13:02.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T17:13:02.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-29T17:13:02.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-29T17:13:02.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.allow_fallocate: 1 2023-07-29T17:13:02.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-29T17:13:02.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.use_direct_reads: 0 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.create_missing_column_families: 0 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.db_log_dir: 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.wal_dir: 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T17:13:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.advise_random_on_open: 1 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.write_buffer_manager: 0x55c69cfc6c80 2023-07-29T17:13:02.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.rate_limiter: (nil) 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-29T17:13:02.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.unordered_write: 0 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.row_cache: None 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.wal_filter: None 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T17:13:02.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.two_write_queues: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.manual_wal_flush: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.wal_compression: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.atomic_flush: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T17:13:02.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.log_readahead_size: 0 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.best_efforts_recovery: 0 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.allow_data_in_errors: 0 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.db_host_id: __hostname__ 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-07-29T17:13:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_background_jobs: 2 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_background_compactions: -1 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_subcompactions: 1 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_total_wal_size: 0 2023-07-29T17:13:02.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_open_files: -1 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bytes_per_sync: 0 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T17:13:02.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_background_flushes: -1 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Compression algorithms supported: 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kZSTD supported: 0 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kXpressCompression supported: 0 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kZlibCompression supported: 1 2023-07-29T17:13:02.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kSnappyCompression supported: 1 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kLZ4Compression supported: 1 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: kBZip2Compression supported: 0 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T17:13:02.705 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.merge_operator: 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_filter: None 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_filter_factory: None 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.sst_partitioner_factory: None 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c69dd39520) 2023-07-29T17:13:02.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: cache_index_and_filter_blocks: 1 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: pin_top_level_index_and_filter: 1 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: index_type: 0 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: data_block_index_type: 0 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: index_shortening: 1 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: data_block_hash_table_util_ratio: 0.750000 2023-07-29T17:13:02.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: checksum: 4 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: no_block_cache: 0 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_cache: 0x55c69d0491f0 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_cache_name: BinnedLRUCache 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_cache_options: 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: capacity : 536870912 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: num_shard_bits : 4 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: strict_capacity_limit : 0 2023-07-29T17:13:02.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: high_pri_pool_ratio: 0.000 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_cache_compressed: (nil) 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: persistent_cache: (nil) 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_size: 4096 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_size_deviation: 10 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_restart_interval: 16 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: index_block_restart_interval: 1 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: metadata_block_size: 4096 2023-07-29T17:13:02.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: partition_filters: 0 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: use_delta_encoding: 1 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: filter_policy: bloomfilter 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: whole_key_filtering: 1 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: verify_compression: 0 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: read_amp_bytes_per_bit: 0 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: format_version: 5 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: enable_index_compression: 1 2023-07-29T17:13:02.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: block_align: 0 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: max_auto_readahead_size: 262144 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: prepopulate_block_cache: 0 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: initial_auto_readahead_size: 8192 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: num_file_reads_for_auto_readahead: 2 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-29T17:13:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression: NoCompression 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.num_levels: 7 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T17:13:02.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-29T17:13:02.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.level: 32767 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-29T17:13:02.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.enabled: false 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T17:13:02.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T17:13:02.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.arena_block_size: 1048576 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T17:13:02.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T17:13:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.table_properties_collectors: 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.inplace_update_support: 0 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T17:13:02.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.bloom_locality: 0 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.max_successive_merges: 0 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.force_consistency_checks: 1 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-29T17:13:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.ttl: 2592000 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.enable_blob_files: false 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.min_blob_size: 0 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.blob_file_size: 268435456 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-07-29T17:13:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.blob_file_starting_level: 0 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cd1abfc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.526+0000 7f0cc72de700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.530+0000 7f0cd1abfc80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-29T17:13:02.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.530+0000 7f0cd1abfc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 11960, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.530+0000 7f0cd1abfc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.530+0000 7f0cd1abfc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9edb6241-ea13-4e19-8f7c-8296584f01ea 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.530+0000 7f0cd1abfc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650782532001, "job": 1, "event": "recovery_started", "wal_files": [18]} 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.530+0000 7f0cd1abfc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.582+0000 7f0cd1abfc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650782585565, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 11664605, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11965, "largest_seqno": 14867, "table_properties": {"data_size": 11649859, "index_size": 11229, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2501, "raw_key_size": 19462, "raw_average_key_size": 19, "raw_value_size": 11624419, "raw_average_value_size": 11671, "num_data_blocks": 494, "num_entries": 996, "num_filter_entries": 996, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1690650782, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9edb6241-ea13-4e19-8f7c-8296584f01ea", "db_session_id": "DWYMMB3GOQ9QTT50GA4D", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.582+0000 7f0cd1abfc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650782585701, "job": 1, "event": "recovery_finished"} 2023-07-29T17:13:02.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.582+0000 7f0cd1abfc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 25 2023-07-29T17:13:02.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.582+0000 7f0cd1abfc80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-29T17:13:02.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.586+0000 7f0cd1abfc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-29T17:13:02.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.586+0000 7f0cd1abfc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c69d0ea000 2023-07-29T17:13:02.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.586+0000 7f0cd1abfc80 4 rocksdb: DB pointer 0x55c69d0d8000 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cc02d0700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cc02d0700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: ** DB Stats ** 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T17:13:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: ** Compaction Stats [default] ** 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: L0 1/0 11.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 572.9 0.02 0.00 1 0.019 0 0 0.0 0.0 2023-07-29T17:13:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: L6 1/0 23.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Sum 2/0 34.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 572.9 0.02 0.00 1 0.019 0 0 0.0 0.0 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 572.9 0.02 0.00 1 0.019 0 0 0.0 0.0 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: ** Compaction Stats [default] ** 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 572.9 0.02 0.00 1 0.019 0 0 0.0 0.0 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-29T17:13:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Flush(GB): cumulative 0.011, interval 0.011 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Cumulative compaction: 0.01 GB write, 180.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Interval compaction: 0.01 GB write, 180.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T17:13:02.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Block cache BinnedLRUCache@0x55c69d0491f0#7 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.4e-05 secs_since: 0 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: Block cache entry stats(count,size,portion): FilterBlock(1,2.53 KB,0.000482798%) IndexBlock(1,12.08 KB,0.00230372%) Misc(1,0.00 KB,0%) 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 starting mon.c rank 2 at public addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] at bind addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 1 mon.c@-1(???) e3 preinit fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 mon.c@-1(???).mds e1 new map 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 mon.c@-1(???).mds e1 print_map 2023-07-29T17:13:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: e1 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: legacy client fscid: -1 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: No filesystems configured 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 mon.c@-1(???).osd e78 crush map has features 3314933000854323200, adjusting msgr requires 2023-07-29T17:13:02.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 mon.c@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:13:02.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 mon.c@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:13:02.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.590+0000 7f0cd1abfc80 0 mon.c@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:13:02.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:02 smithi078 bash[60360]: debug 2023-07-29T17:13:02.594+0000 7f0cd1abfc80 1 mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2023-07-29T17:13:08.145 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:13:08.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:07 smithi129 bash[23526]: cluster 2023-07-29T17:13:01.144732+0000 mgr.y (mgr.25169) 27 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:08.889 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:13:08.890 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3m) 28s ago 9m 10.2M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:13:08.890 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (2m) 27s ago 8m 25.3M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:13:08.890 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 27s ago 6m - - 2023-07-29T17:13:08.890 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (72s) 27s ago 13m 426M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:13:08.890 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (2m) 28s ago 15m 456M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:13:08.890 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (46s) 28s ago 15m 33.6M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (14m) 27s ago 14m 63.3M 2048M 16.2.0 24ecd6d5f14c 46de86e9cc63 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 starting - - - 2048M 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3m) 28s ago 9m 8987k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (2m) 27s ago 9m 3636k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (13m) 28s ago 13m 55.7M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (12m) 28s ago 12m 53.1M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (12m) 28s ago 12m 47.9M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:13:08.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (11m) 28s ago 11m 53.5M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (11m) 27s ago 11m 54.8M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (11m) 27s ago 11m 52.8M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (10m) 27s ago 10m 53.1M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (10m) 27s ago 10m 56.3M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 27s ago 9m - - 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (7m) 28s ago 7m 53.8M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:13:08.892 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (7m) 27s ago 7m 53.9M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:13:08.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (7m) 28s ago 7m 54.0M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:13:08.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (7m) 27s ago 7m 53.9M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:13:09.334 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:13:09.334 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:13:09.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:13:09.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 13, 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:13:09.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:13:09.337 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:13:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:13:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:13:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "3/23 daemons upgraded", 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:13:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:13:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:02.809530+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2023-07-29T17:13:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: audit 2023-07-29T17:13:08.492313+0000 mgr.y (mgr.25169) 32 : audit [DBG] from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.815214+0000 mon.a (mon.0) 83 : cluster [INF] mon.a calling monitor election 2023-07-29T17:13:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.816454+0000 mon.b (mon.1) 109 : cluster [INF] mon.b calling monitor election 2023-07-29T17:13:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.819527+0000 mon.a (mon.0) 84 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:09.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.835046+0000 mon.a (mon.0) 85 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.835476+0000 mon.a (mon.0) 86 : cluster [DBG] fsmap 2023-07-29T17:13:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.835512+0000 mon.a (mon.0) 87 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.835890+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e31: y(active, since 35s), standbys: x 2023-07-29T17:13:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.836038+0000 mon.a (mon.0) 89 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T17:13:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.839644+0000 mon.a (mon.0) 90 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:09.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.839667+0000 mon.a (mon.0) 91 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.839675+0000 mon.a (mon.0) 92 : cluster [WRN] daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: cluster 2023-07-29T17:13:08.839682+0000 mon.a (mon.0) 93 : cluster [WRN] daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: audit 2023-07-29T17:13:08.852283+0000 mgr.y (mgr.25169) 33 : audit [DBG] from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: audit 2023-07-29T17:13:09.014463+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.129:0/3636614956' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:09.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:09 smithi129 bash[23526]: audit 2023-07-29T17:13:09.315234+0000 mon.a (mon.0) 94 : audit [DBG] from='client.34254 172.21.15.78:0/4088428933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:10.146 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:10.146 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:10.146 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:10.146 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:11.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:10 smithi129 bash[23526]: cluster 2023-07-29T17:13:09.148744+0000 mgr.y (mgr.25169) 34 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:11.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:10 smithi129 bash[23526]: audit 2023-07-29T17:13:09.693269+0000 mgr.y (mgr.25169) 35 : audit [DBG] from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:11.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:10 smithi129 bash[23526]: audit 2023-07-29T17:13:10.127408+0000 mon.a (mon.0) 95 : audit [DBG] from='client.34266 172.21.15.78:0/2581941085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:12.940 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:13:12 smithi078 bash[55873]: debug 2023-07-29T17:13:12.601+0000 7f900e65c700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-29T17:13:13.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:12 smithi129 bash[23526]: cluster 2023-07-29T17:13:11.150593+0000 mgr.y (mgr.25169) 36 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:15.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:14 smithi129 bash[23526]: cluster 2023-07-29T17:13:13.151433+0000 mgr.y (mgr.25169) 37 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:15.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:14 smithi129 bash[23526]: audit 2023-07-29T17:13:14.413473+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:15.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:14 smithi129 bash[23526]: audit 2023-07-29T17:13:14.425544+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:16.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:16 smithi129 bash[23526]: cluster 2023-07-29T17:13:15.153124+0000 mgr.y (mgr.25169) 38 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:16.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:16 smithi129 bash[23526]: audit 2023-07-29T17:13:15.223048+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:16.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:16 smithi129 bash[23526]: audit 2023-07-29T17:13:15.234260+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:18.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:18 smithi129 bash[23526]: cluster 2023-07-29T17:13:17.153885+0000 mgr.y (mgr.25169) 39 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:20 smithi129 bash[23526]: cluster 2023-07-29T17:13:19.155285+0000 mgr.y (mgr.25169) 40 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:22.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:22 smithi129 bash[23526]: cluster 2023-07-29T17:13:21.156870+0000 mgr.y (mgr.25169) 41 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:25.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:24 smithi129 bash[23526]: cluster 2023-07-29T17:13:23.157602+0000 mgr.y (mgr.25169) 42 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:26.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: cluster 2023-07-29T17:13:25.159335+0000 mgr.y (mgr.25169) 43 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:26.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.486193+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:26.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.497119+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:26.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.499021+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:26.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.500542+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:26.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: cephadm 2023-07-29T17:13:25.501923+0000 mgr.y (mgr.25169) 44 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: cluster 2023-07-29T17:13:25.503192+0000 mgr.y (mgr.25169) 45 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.511989+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.600876+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.605319+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.607573+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: audit 2023-07-29T17:13:25.608919+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-29T17:13:26.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:26 smithi129 bash[23526]: cephadm 2023-07-29T17:13:25.610067+0000 mgr.y (mgr.25169) 46 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2023-07-29T17:13:28.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:28 smithi129 bash[23526]: cluster 2023-07-29T17:13:27.503937+0000 mgr.y (mgr.25169) 47 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:28.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:28 smithi129 bash[23526]: audit 2023-07-29T17:13:28.191152+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:28.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:28 smithi129 bash[23526]: audit 2023-07-29T17:13:28.192848+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:28.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:28 smithi129 bash[23526]: audit 2023-07-29T17:13:28.194395+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:28.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:28 smithi129 bash[23526]: audit 2023-07-29T17:13:28.195803+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:29.828 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:29 smithi129 bash[23526]: cephadm 2023-07-29T17:13:28.181361+0000 mgr.y (mgr.25169) 48 : cephadm [INF] Upgrade: Updating mon.b 2023-07-29T17:13:29.828 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:29 smithi129 bash[23526]: cephadm 2023-07-29T17:13:28.197401+0000 mgr.y (mgr.25169) 49 : cephadm [INF] Deploying daemon mon.b on smithi129 2023-07-29T17:13:30.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:30 smithi129 bash[23526]: cluster 2023-07-29T17:13:29.505619+0000 mgr.y (mgr.25169) 50 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:30.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:30 smithi129 systemd[1]: Stopping Ceph mon.b for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:13:31.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:30 smithi129 bash[23526]: debug 2023-07-29T17:13:30.805+0000 7f1e02ac7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T17:13:31.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:30 smithi129 bash[23526]: debug 2023-07-29T17:13:30.805+0000 7f1e02ac7700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2023-07-29T17:13:32.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:31 smithi129 bash[53591]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-mon.b 2023-07-29T17:13:32.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:31 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.b.service: Succeeded. 2023-07-29T17:13:32.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:31 smithi129 systemd[1]: Stopped Ceph mon.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:13:32.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:32 smithi129 systemd[1]: Started Ceph mon.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:13:33.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.221+0000 7f5540a65c80 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-29T17:13:33.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.221+0000 7f5540a65c80 0 ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc), process ceph-mon, pid 7 2023-07-29T17:13:33.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.221+0000 7f5540a65c80 0 pidfile_write: ignore empty --pid-file 2023-07-29T17:13:33.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 0 load: jerasure load: lrc 2023-07-29T17:13:33.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: RocksDB version: 7.9.2 2023-07-29T17:13:33.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Git sha 0 2023-07-29T17:13:33.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Compile date 2023-07-28 16:56:26 2023-07-29T17:13:33.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: DB SUMMARY 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: DB Session ID: BUEH6XDMKF1SUD6R7AFK 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: CURRENT file: CURRENT 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: IDENTITY file: IDENTITY 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 882 Bytes 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000020.sst 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000018.log size: 20201411 ; 2023-07-29T17:13:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.error_if_exists: 0 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.create_if_missing: 0 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.paranoid_checks: 1 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.env: 0x55db9e90e720 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.fs: PosixFileSystem 2023-07-29T17:13:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.info_log: 0x55dba02953e0 2023-07-29T17:13:33.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-29T17:13:33.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.statistics: (nil) 2023-07-29T17:13:33.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.use_fsync: 0 2023-07-29T17:13:33.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_log_file_size: 0 2023-07-29T17:13:33.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T17:13:33.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.allow_fallocate: 1 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.use_direct_reads: 0 2023-07-29T17:13:33.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T17:13:33.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.create_missing_column_families: 0 2023-07-29T17:13:33.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.db_log_dir: 2023-07-29T17:13:33.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.wal_dir: 2023-07-29T17:13:33.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T17:13:33.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T17:13:33.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T17:13:33.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T17:13:33.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T17:13:33.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T17:13:33.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.advise_random_on_open: 1 2023-07-29T17:13:33.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-29T17:13:33.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.write_buffer_manager: 0x55db9f522c80 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.rate_limiter: (nil) 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-29T17:13:33.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.unordered_write: 0 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T17:13:33.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.row_cache: None 2023-07-29T17:13:33.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.wal_filter: None 2023-07-29T17:13:33.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T17:13:33.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-29T17:13:33.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.two_write_queues: 0 2023-07-29T17:13:33.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.manual_wal_flush: 0 2023-07-29T17:13:33.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.wal_compression: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.atomic_flush: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.log_readahead_size: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.best_efforts_recovery: 0 2023-07-29T17:13:33.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.allow_data_in_errors: 0 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.db_host_id: __hostname__ 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_background_jobs: 2 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_background_compactions: -1 2023-07-29T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_subcompactions: 1 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_total_wal_size: 0 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T17:13:33.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_open_files: -1 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bytes_per_sync: 0 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_background_flushes: -1 2023-07-29T17:13:33.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Compression algorithms supported: 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kZSTD supported: 0 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kXpressCompression supported: 0 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kZlibCompression supported: 1 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kSnappyCompression supported: 1 2023-07-29T17:13:33.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kLZ4Compression supported: 1 2023-07-29T17:13:33.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: kBZip2Compression supported: 0 2023-07-29T17:13:33.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T17:13:33.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-07-29T17:13:33.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-29T17:13:33.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-29T17:13:33.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T17:13:33.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.merge_operator: 2023-07-29T17:13:33.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_filter: None 2023-07-29T17:13:33.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_filter_factory: None 2023-07-29T17:13:33.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.sst_partitioner_factory: None 2023-07-29T17:13:33.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T17:13:33.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T17:13:33.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dba0295520) 2023-07-29T17:13:33.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: cache_index_and_filter_blocks: 1 2023-07-29T17:13:33.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T17:13:33.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T17:13:33.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: pin_top_level_index_and_filter: 1 2023-07-29T17:13:33.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: index_type: 0 2023-07-29T17:13:33.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: data_block_index_type: 0 2023-07-29T17:13:33.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: index_shortening: 1 2023-07-29T17:13:33.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: data_block_hash_table_util_ratio: 0.750000 2023-07-29T17:13:33.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: checksum: 4 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: no_block_cache: 0 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_cache: 0x55db9f5a51f0 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_cache_name: BinnedLRUCache 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_cache_options: 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: capacity : 536870912 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: num_shard_bits : 4 2023-07-29T17:13:33.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: strict_capacity_limit : 0 2023-07-29T17:13:33.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: high_pri_pool_ratio: 0.000 2023-07-29T17:13:33.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_cache_compressed: (nil) 2023-07-29T17:13:33.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: persistent_cache: (nil) 2023-07-29T17:13:33.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_size: 4096 2023-07-29T17:13:33.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_size_deviation: 10 2023-07-29T17:13:33.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_restart_interval: 16 2023-07-29T17:13:33.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: index_block_restart_interval: 1 2023-07-29T17:13:33.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: metadata_block_size: 4096 2023-07-29T17:13:33.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: partition_filters: 0 2023-07-29T17:13:33.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: use_delta_encoding: 1 2023-07-29T17:13:33.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: filter_policy: bloomfilter 2023-07-29T17:13:33.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: whole_key_filtering: 1 2023-07-29T17:13:33.447 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: verify_compression: 0 2023-07-29T17:13:33.447 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: read_amp_bytes_per_bit: 0 2023-07-29T17:13:33.447 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: format_version: 5 2023-07-29T17:13:33.447 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: enable_index_compression: 1 2023-07-29T17:13:33.447 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: block_align: 0 2023-07-29T17:13:33.447 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: max_auto_readahead_size: 262144 2023-07-29T17:13:33.448 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: prepopulate_block_cache: 0 2023-07-29T17:13:33.448 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: initial_auto_readahead_size: 8192 2023-07-29T17:13:33.448 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: num_file_reads_for_auto_readahead: 2 2023-07-29T17:13:33.448 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-29T17:13:33.448 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression: NoCompression 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.num_levels: 7 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T17:13:33.449 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-29T17:13:33.451 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-29T17:13:33.451 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T17:13:33.451 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.level: 32767 2023-07-29T17:13:33.451 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-29T17:13:33.451 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T17:13:33.451 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.enabled: false 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T17:13:33.452 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T17:13:33.453 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-29T17:13:33.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.arena_block_size: 1048576 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T17:13:33.455 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.table_properties_collectors: 2023-07-29T17:13:33.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.inplace_update_support: 0 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.bloom_locality: 0 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.max_successive_merges: 0 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T17:13:33.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.force_consistency_checks: 1 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.ttl: 2592000 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.enable_blob_files: false 2023-07-29T17:13:33.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.min_blob_size: 0 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.blob_file_size: 268435456 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-29T17:13:33.459 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.blob_file_starting_level: 0 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5540a65c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.225+0000 7f5536284700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.229+0000 7f5540a65c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.229+0000 7f5540a65c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12023, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.229+0000 7f5540a65c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.229+0000 7f5540a65c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d6ff7980-3082-4250-a7b5-dbbed3f4226b 2023-07-29T17:13:33.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.229+0000 7f5540a65c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650813231954, "job": 1, "event": "recovery_started", "wal_files": [18]} 2023-07-29T17:13:33.461 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.229+0000 7f5540a65c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2023-07-29T17:13:33.461 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.285+0000 7f5540a65c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650813287555, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 12955069, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12028, "largest_seqno": 15332, "table_properties": {"data_size": 12938583, "index_size": 12713, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2757, "raw_key_size": 21273, "raw_average_key_size": 19, "raw_value_size": 12910378, "raw_average_value_size": 11822, "num_data_blocks": 559, "num_entries": 1092, "num_filter_entries": 1092, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1690650813, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d6ff7980-3082-4250-a7b5-dbbed3f4226b", "db_session_id": "BUEH6XDMKF1SUD6R7AFK", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2023-07-29T17:13:33.461 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.285+0000 7f5540a65c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1690650813287771, "job": 1, "event": "recovery_finished"} 2023-07-29T17:13:33.461 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.285+0000 7f5540a65c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 25 2023-07-29T17:13:33.461 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.285+0000 7f5540a65c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.289+0000 7f5540a65c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.289+0000 7f5540a65c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55db9f646000 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.289+0000 7f5540a65c80 4 rocksdb: DB pointer 0x55db9f634000 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.289+0000 7f552f276700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.289+0000 7f552f276700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: ** DB Stats ** 2023-07-29T17:13:33.462 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: ** Compaction Stats [default] ** 2023-07-29T17:13:33.463 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: L0 1/0 12.35 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 564.7 0.02 0.00 1 0.022 0 0 0.0 0.0 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: L6 1/0 23.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Sum 2/0 35.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 564.7 0.02 0.00 1 0.022 0 0 0.0 0.0 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 564.7 0.02 0.00 1 0.022 0 0 0.0 0.0 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: ** Compaction Stats [default] ** 2023-07-29T17:13:33.464 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 564.7 0.02 0.00 1 0.022 0 0 0.0 0.0 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Flush(GB): cumulative 0.012, interval 0.012 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T17:13:33.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: AddFile(Keys): cumulative 0, interval 0 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Cumulative compaction: 0.01 GB write, 196.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Interval compaction: 0.01 GB write, 196.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Block cache BinnedLRUCache@0x55db9f5a51f0#7 capacity: 512.00 MB usage: 15.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: Block cache entry stats(count,size,portion): FilterBlock(1,2.78 KB,0.000530481%) IndexBlock(1,13.08 KB,0.00249445%) Misc(1,0.00 KB,0%) 2023-07-29T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: ** File Read Latency Histogram By Level [default] ** 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.289+0000 7f5540a65c80 0 starting mon.b rank 1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 1 mon.b@-1(???) e3 preinit fsid eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 0 mon.b@-1(???).mds e1 new map 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 0 mon.b@-1(???).mds e1 print_map 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: e1 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T17:13:33.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: legacy client fscid: -1 2023-07-29T17:13:33.468 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: 2023-07-29T17:13:33.468 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: No filesystems configured 2023-07-29T17:13:33.468 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 0 mon.b@-1(???).osd e78 crush map has features 3314933000854323200, adjusting msgr requires 2023-07-29T17:13:33.468 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 0 mon.b@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:13:33.468 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 0 mon.b@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:13:33.468 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.293+0000 7f5540a65c80 0 mon.b@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2023-07-29T17:13:33.469 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:33 smithi129 bash[53756]: debug 2023-07-29T17:13:33.297+0000 7f5540a65c80 1 mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2023-07-29T17:13:35.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T16:57:39.388557+0000 mon.a (mon.0) 0 : 1 mkfs eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:35.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:09:53.860450+0000 mgr.x (mgr.24745) 25 : cephadm 1 Reconfiguring node-exporter.a (dependencies changed)... 2023-07-29T17:13:35.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:09:53.861126+0000 mgr.x (mgr.24745) 26 : cephadm 1 Reconfiguring daemon node-exporter.a on smithi078 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:09:55.078863+0000 mgr.x (mgr.24745) 30 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:09:55.100231+0000 mgr.x (mgr.24745) 31 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:04.591366+0000 mgr.x (mgr.24745) 38 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:04.832178+0000 mgr.x (mgr.24745) 39 : cephadm 1 Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:19.033761+0000 mgr.x (mgr.24745) 47 : cephadm 1 Reconfiguring node-exporter.b (dependencies changed)... 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:19.034250+0000 mgr.x (mgr.24745) 48 : cephadm 1 Reconfiguring daemon node-exporter.b on smithi129 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:20.803607+0000 mgr.x (mgr.24745) 50 : cephadm 1 Reconfiguring grafana.a (dependencies changed)... 2023-07-29T17:13:35.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:20.819379+0000 mgr.x (mgr.24745) 51 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:20.932157+0000 mgr.x (mgr.24745) 53 : cephadm 1 Reconfiguring daemon grafana.a on smithi129 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:47.252030+0000 mgr.x (mgr.24745) 71 : cephadm 1 Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:47.266606+0000 mgr.x (mgr.24745) 72 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:54.194148+0000 mgr.x (mgr.24745) 84 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:54.195301+0000 mgr.x (mgr.24745) 85 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:56.415335+0000 mgr.x (mgr.24745) 88 : cephadm 1 Upgrade: Updating mgr.y 2023-07-29T17:13:35.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:10:56.433996+0000 mgr.x (mgr.24745) 89 : cephadm 1 Deploying daemon mgr.y on smithi078 2023-07-29T17:13:35.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:18.568228+0000 mgr.x (mgr.24745) 105 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:18.669628+0000 mgr.x (mgr.24745) 107 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:18.670428+0000 mgr.x (mgr.24745) 108 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:18.673567+0000 mgr.x (mgr.24745) 109 : cephadm 1 Failing over to other MGR 2023-07-29T17:13:35.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:19.668775+0000 mgr.y (mgr.15066) 1 : cephadm 1 inventory: adjusted host smithi129 addr 'smithi129' -> '172.21.15.129' 2023-07-29T17:13:35.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:20.194974+0000 mgr.y (mgr.15066) 2 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Bus STARTING 2023-07-29T17:13:35.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:20.317199+0000 mgr.y (mgr.15066) 3 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:13:35.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:20.419606+0000 mgr.y (mgr.15066) 4 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:20.420293+0000 mgr.y (mgr.15066) 5 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Bus STARTED 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:37.901163+0000 mgr.y (mgr.15066) 20 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:37.984920+0000 mgr.y (mgr.15066) 22 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:38.232589+0000 mgr.y (mgr.15066) 23 : cephadm 1 Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:42.914135+0000 mgr.y (mgr.15066) 26 : cephadm 1 Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:42.929361+0000 mgr.y (mgr.15066) 27 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:13:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:51.152326+0000 mgr.y (mgr.15066) 36 : cephadm 1 Upgrade: Updating mgr.x 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:11:51.159128+0000 mgr.y (mgr.15066) 37 : cephadm 1 Deploying daemon mgr.x on smithi129 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:16.269542+0000 mgr.y (mgr.15066) 55 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:16.373930+0000 mgr.y (mgr.15066) 58 : cephadm 1 Upgrade: Setting container_image for all mgr 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:16.410372+0000 mgr.y (mgr.15066) 59 : cephadm 1 Upgrade: It appears safe to stop mon.a 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:18.630942+0000 mgr.y (mgr.15066) 61 : cephadm 1 Upgrade: Updating mon.a 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:18.648913+0000 mgr.y (mgr.15066) 62 : cephadm 1 Deploying daemon mon.a on smithi078 2023-07-29T17:13:35.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:33.139239+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:33.225936+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:33.352681+0000 mon.a (mon.0) 38 : cluster 0 Standby manager daemon x restarted 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:33.352894+0000 mon.a (mon.0) 39 : cluster 0 Standby manager daemon x started 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:33.691763+0000 mgr.y (mgr.25169) 1 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Bus STARTING 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:33.818280+0000 mgr.y (mgr.25169) 2 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:33.851254+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:33.920427+0000 mgr.y (mgr.25169) 3 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:13:35.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:33.921157+0000 mgr.y (mgr.25169) 4 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Bus STARTED 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:34.140406+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e30: y(active, since 1.02058s), standbys: x 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:34.155361+0000 mgr.y (mgr.25169) 5 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:35.132649+0000 mon.a (mon.0) 41 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:35.132713+0000 mon.a (mon.0) 42 : cluster 1 Cluster is now healthy 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:35.155397+0000 mgr.y (mgr.25169) 6 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:36.151205+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e31: y(active, since 3s), standbys: x 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:37.156085+0000 mgr.y (mgr.25169) 10 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:37.270148+0000 mgr.y (mgr.25169) 11 : audit 0 from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:37.843909+0000 mon.b (mon.1) 104 : audit 1 from='client.? 172.21.15.129:0/158598556' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:39.156986+0000 mgr.y (mgr.25169) 12 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:40.182021+0000 mon.a (mon.0) 44 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:40.192107+0000 mon.a (mon.0) 45 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:40.893782+0000 mon.a (mon.0) 46 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:40.903310+0000 mon.a (mon.0) 47 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:40.990950+0000 mon.a (mon.0) 48 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:41.002809+0000 mon.a (mon.0) 49 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:41.157681+0000 mgr.y (mgr.25169) 13 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:41.701420+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:41.712500+0000 mon.a (mon.0) 51 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:43.158401+0000 mgr.y (mgr.25169) 14 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:45.160084+0000 mgr.y (mgr.25169) 15 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:47.160880+0000 mgr.y (mgr.25169) 16 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:49.162452+0000 mgr.y (mgr.25169) 17 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:51.163203+0000 mgr.y (mgr.25169) 18 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:51.171698+0000 mon.a (mon.0) 52 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:51.181721+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:53.163956+0000 mgr.y (mgr.25169) 19 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:53.373130+0000 mon.b (mon.1) 105 : audit 0 from='client.? 172.21.15.129:0/720231089' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:53.744700+0000 mon.b (mon.1) 106 : audit 1 from='client.? 172.21.15.129:0/2122394065' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.125639+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.135001+0000 mon.a (mon.0) 55 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.137619+0000 mon.a (mon.0) 56 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.139021+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:55.140241+0000 mgr.y (mgr.25169) 20 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:55.141962+0000 mgr.y (mgr.25169) 21 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.147652+0000 mon.a (mon.0) 58 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.234039+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:35.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.238749+0000 mon.a (mon.0) 60 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.242123+0000 mon.a (mon.0) 61 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:55.244043+0000 mon.a (mon.0) 62 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:55.245469+0000 mgr.y (mgr.25169) 22 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:56.131866+0000 mon.a (mon.0) 63 : cluster 3 Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:57.142651+0000 mgr.y (mgr.25169) 23 : cluster 0 pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:57.514548+0000 mgr.y (mgr.25169) 24 : cephadm 1 Upgrade: Updating mon.c 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:57.524116+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.435 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:57.525945+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:57.527053+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:12:57.528224+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:12:57.529886+0000 mgr.y (mgr.25169) 25 : cephadm 1 Deploying daemon mon.c on smithi078 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:12:59.143567+0000 mgr.y (mgr.25169) 26 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:01.144732+0000 mgr.y (mgr.25169) 27 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:02.809530+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2023-07-29T17:13:35.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:08.492313+0000 mgr.y (mgr.25169) 32 : audit 0 from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.815214+0000 mon.a (mon.0) 83 : cluster 1 mon.a calling monitor election 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.816454+0000 mon.b (mon.1) 109 : cluster 1 mon.b calling monitor election 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.819527+0000 mon.a (mon.0) 84 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.835046+0000 mon.a (mon.0) 85 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.835476+0000 mon.a (mon.0) 86 : cluster 0 fsmap 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.835512+0000 mon.a (mon.0) 87 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.835890+0000 mon.a (mon.0) 88 : cluster 0 mgrmap e31: y(active, since 35s), standbys: x 2023-07-29T17:13:35.437 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.836038+0000 mon.a (mon.0) 89 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.839644+0000 mon.a (mon.0) 90 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.839667+0000 mon.a (mon.0) 91 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.839675+0000 mon.a (mon.0) 92 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:08.839682+0000 mon.a (mon.0) 93 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:08.852283+0000 mgr.y (mgr.25169) 33 : audit 0 from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:09.014463+0000 mon.b (mon.1) 110 : audit 1 from='client.? 172.21.15.129:0/3636614956' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:09.148744+0000 mgr.y (mgr.25169) 34 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:09.315234+0000 mon.a (mon.0) 94 : audit 0 from='client.34254 172.21.15.78:0/4088428933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:09.693269+0000 mgr.y (mgr.25169) 35 : audit 0 from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:10.127408+0000 mon.a (mon.0) 95 : audit 0 from='client.34266 172.21.15.78:0/2581941085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:11.150593+0000 mgr.y (mgr.25169) 36 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:13.151433+0000 mgr.y (mgr.25169) 37 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:14.413473+0000 mon.a (mon.0) 96 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:14.425544+0000 mon.a (mon.0) 97 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:15.153124+0000 mgr.y (mgr.25169) 38 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:15.223048+0000 mon.a (mon.0) 98 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:15.234260+0000 mon.a (mon.0) 99 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:17.153885+0000 mgr.y (mgr.25169) 39 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:19.155285+0000 mgr.y (mgr.25169) 40 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:21.156870+0000 mgr.y (mgr.25169) 41 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:23.157602+0000 mgr.y (mgr.25169) 42 : cluster 0 pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:25.159335+0000 mgr.y (mgr.25169) 43 : cluster 0 pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.440 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.486193+0000 mon.a (mon.0) 100 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.497119+0000 mon.a (mon.0) 101 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.499021+0000 mon.a (mon.0) 102 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.500542+0000 mon.a (mon.0) 103 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:13:25.501923+0000 mgr.y (mgr.25169) 44 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:25.503192+0000 mgr.y (mgr.25169) 45 : cluster 0 pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.511989+0000 mon.a (mon.0) 104 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.600876+0000 mon.a (mon.0) 105 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.605319+0000 mon.a (mon.0) 106 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.607573+0000 mon.a (mon.0) 107 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:25.608919+0000 mon.a (mon.0) 108 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:13:25.610067+0000 mgr.y (mgr.25169) 46 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:27.503937+0000 mgr.y (mgr.25169) 47 : cluster 0 pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:13:28.181361+0000 mgr.y (mgr.25169) 48 : cephadm 1 Upgrade: Updating mon.b 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:28.191152+0000 mon.a (mon.0) 109 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.442 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:28.192848+0000 mon.a (mon.0) 110 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:28.194395+0000 mon.a (mon.0) 111 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:28.195803+0000 mon.a (mon.0) 112 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cephadm 2023-07-29T17:13:28.197401+0000 mgr.y (mgr.25169) 49 : cephadm 1 Deploying daemon mon.b on smithi129 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:29.505619+0000 mgr.y (mgr.25169) 50 : cluster 0 pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:33.507127+0000 mgr.y (mgr.25169) 52 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.094676+0000 mon.a (mon.0) 119 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.443 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.094705+0000 mon.a (mon.0) 120 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.094718+0000 mon.a (mon.0) 121 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.094733+0000 mon.a (mon.0) 122 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:34.095259+0000 mon.a (mon.0) 123 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.095780+0000 mon.a (mon.0) 124 : cluster 1 mon.a calling monitor election 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.095971+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:34.097385+0000 mon.a (mon.0) 125 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:34.097611+0000 mon.a (mon.0) 126 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:13:35.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.098256+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.099483+0000 mon.a (mon.0) 127 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.107867+0000 mon.a (mon.0) 128 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.108830+0000 mon.a (mon.0) 129 : cluster 0 fsmap 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.108896+0000 mon.a (mon.0) 130 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.109554+0000 mon.a (mon.0) 131 : cluster 0 mgrmap e31: y(active, since 60s), standbys: x 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.114353+0000 mon.a (mon.0) 132 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.114388+0000 mon.a (mon.0) 133 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.114402+0000 mon.a (mon.0) 134 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: cluster 2023-07-29T17:13:34.114433+0000 mon.a (mon.0) 135 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:34.117832+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:35 smithi129 bash[53756]: audit 2023-07-29T17:13:34.126071+0000 mon.a (mon.0) 137 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T16:57:39.388557+0000 mon.a (mon.0) 0 : 1 mkfs eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:09:53.860450+0000 mgr.x (mgr.24745) 25 : cephadm 1 Reconfiguring node-exporter.a (dependencies changed)... 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:09:53.861126+0000 mgr.x (mgr.24745) 26 : cephadm 1 Reconfiguring daemon node-exporter.a on smithi078 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:09:55.078863+0000 mgr.x (mgr.24745) 30 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:09:55.100231+0000 mgr.x (mgr.24745) 31 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:04.591366+0000 mgr.x (mgr.24745) 38 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:04.832178+0000 mgr.x (mgr.24745) 39 : cephadm 1 Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:13:35.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:19.033761+0000 mgr.x (mgr.24745) 47 : cephadm 1 Reconfiguring node-exporter.b (dependencies changed)... 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:19.034250+0000 mgr.x (mgr.24745) 48 : cephadm 1 Reconfiguring daemon node-exporter.b on smithi129 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:20.803607+0000 mgr.x (mgr.24745) 50 : cephadm 1 Reconfiguring grafana.a (dependencies changed)... 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:20.819379+0000 mgr.x (mgr.24745) 51 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:20.932157+0000 mgr.x (mgr.24745) 53 : cephadm 1 Reconfiguring daemon grafana.a on smithi129 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:47.252030+0000 mgr.x (mgr.24745) 71 : cephadm 1 Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:47.266606+0000 mgr.x (mgr.24745) 72 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:54.194148+0000 mgr.x (mgr.24745) 84 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:54.195301+0000 mgr.x (mgr.24745) 85 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:56.415335+0000 mgr.x (mgr.24745) 88 : cephadm 1 Upgrade: Updating mgr.y 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:10:56.433996+0000 mgr.x (mgr.24745) 89 : cephadm 1 Deploying daemon mgr.y on smithi078 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:18.568228+0000 mgr.x (mgr.24745) 105 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:18.669628+0000 mgr.x (mgr.24745) 107 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:18.670428+0000 mgr.x (mgr.24745) 108 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:18.673567+0000 mgr.x (mgr.24745) 109 : cephadm 1 Failing over to other MGR 2023-07-29T17:13:35.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:19.668775+0000 mgr.y (mgr.15066) 1 : cephadm 1 inventory: adjusted host smithi129 addr 'smithi129' -> '172.21.15.129' 2023-07-29T17:13:35.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:20.194974+0000 mgr.y (mgr.15066) 2 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Bus STARTING 2023-07-29T17:13:35.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T16:57:39.388557+0000 mon.a (mon.0) 0 : 1 mkfs eb1107b2-2e30-11ee-9b36-001a4aab830c 2023-07-29T17:13:35.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:09:53.860450+0000 mgr.x (mgr.24745) 25 : cephadm 1 Reconfiguring node-exporter.a (dependencies changed)... 2023-07-29T17:13:35.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:09:53.861126+0000 mgr.x (mgr.24745) 26 : cephadm 1 Reconfiguring daemon node-exporter.a on smithi078 2023-07-29T17:13:35.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:09:55.078863+0000 mgr.x (mgr.24745) 30 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2023-07-29T17:13:35.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:09:55.100231+0000 mgr.x (mgr.24745) 31 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi078 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:04.591366+0000 mgr.x (mgr.24745) 38 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:04.832178+0000 mgr.x (mgr.24745) 39 : cephadm 1 Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:19.033761+0000 mgr.x (mgr.24745) 47 : cephadm 1 Reconfiguring node-exporter.b (dependencies changed)... 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:19.034250+0000 mgr.x (mgr.24745) 48 : cephadm 1 Reconfiguring daemon node-exporter.b on smithi129 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:20.803607+0000 mgr.x (mgr.24745) 50 : cephadm 1 Reconfiguring grafana.a (dependencies changed)... 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:20.819379+0000 mgr.x (mgr.24745) 51 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:20.932157+0000 mgr.x (mgr.24745) 53 : cephadm 1 Reconfiguring daemon grafana.a on smithi129 2023-07-29T17:13:35.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:47.252030+0000 mgr.x (mgr.24745) 71 : cephadm 1 Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:47.266606+0000 mgr.x (mgr.24745) 72 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:54.194148+0000 mgr.x (mgr.24745) 84 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:54.195301+0000 mgr.x (mgr.24745) 85 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:56.415335+0000 mgr.x (mgr.24745) 88 : cephadm 1 Upgrade: Updating mgr.y 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:10:56.433996+0000 mgr.x (mgr.24745) 89 : cephadm 1 Deploying daemon mgr.y on smithi078 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:18.568228+0000 mgr.x (mgr.24745) 105 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:18.669628+0000 mgr.x (mgr.24745) 107 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:18.670428+0000 mgr.x (mgr.24745) 108 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-29T17:13:35.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:18.673567+0000 mgr.x (mgr.24745) 109 : cephadm 1 Failing over to other MGR 2023-07-29T17:13:35.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:19.668775+0000 mgr.y (mgr.15066) 1 : cephadm 1 inventory: adjusted host smithi129 addr 'smithi129' -> '172.21.15.129' 2023-07-29T17:13:35.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:20.194974+0000 mgr.y (mgr.15066) 2 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Bus STARTING 2023-07-29T17:13:35.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:20.317199+0000 mgr.y (mgr.15066) 3 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:13:35.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:20.419606+0000 mgr.y (mgr.15066) 4 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:20.420293+0000 mgr.y (mgr.15066) 5 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Bus STARTED 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:37.901163+0000 mgr.y (mgr.15066) 20 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:37.984920+0000 mgr.y (mgr.15066) 22 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:38.232589+0000 mgr.y (mgr.15066) 23 : cephadm 1 Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:42.914135+0000 mgr.y (mgr.15066) 26 : cephadm 1 Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:42.929361+0000 mgr.y (mgr.15066) 27 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:13:35.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:51.152326+0000 mgr.y (mgr.15066) 36 : cephadm 1 Upgrade: Updating mgr.x 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:11:51.159128+0000 mgr.y (mgr.15066) 37 : cephadm 1 Deploying daemon mgr.x on smithi129 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:16.269542+0000 mgr.y (mgr.15066) 55 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:16.373930+0000 mgr.y (mgr.15066) 58 : cephadm 1 Upgrade: Setting container_image for all mgr 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:16.410372+0000 mgr.y (mgr.15066) 59 : cephadm 1 Upgrade: It appears safe to stop mon.a 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:18.630942+0000 mgr.y (mgr.15066) 61 : cephadm 1 Upgrade: Updating mon.a 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:18.648913+0000 mgr.y (mgr.15066) 62 : cephadm 1 Deploying daemon mon.a on smithi078 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:33.139239+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:33.225936+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:33.352681+0000 mon.a (mon.0) 38 : cluster 0 Standby manager daemon x restarted 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:33.352894+0000 mon.a (mon.0) 39 : cluster 0 Standby manager daemon x started 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:33.691763+0000 mgr.y (mgr.25169) 1 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Bus STARTING 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:33.818280+0000 mgr.y (mgr.25169) 2 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:33.851254+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:33.920427+0000 mgr.y (mgr.25169) 3 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:33.921157+0000 mgr.y (mgr.25169) 4 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Bus STARTED 2023-07-29T17:13:35.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:34.140406+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e30: y(active, since 1.02058s), standbys: x 2023-07-29T17:13:35.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:20.317199+0000 mgr.y (mgr.15066) 3 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:13:35.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:20.419606+0000 mgr.y (mgr.15066) 4 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:13:35.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:20.420293+0000 mgr.y (mgr.15066) 5 : cephadm 1 [29/Jul/2023:17:11:20] ENGINE Bus STARTED 2023-07-29T17:13:35.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:37.901163+0000 mgr.y (mgr.15066) 20 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:37.984920+0000 mgr.y (mgr.15066) 22 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-29T17:13:35.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:38.232589+0000 mgr.y (mgr.15066) 23 : cephadm 1 Reconfiguring daemon prometheus.a on smithi129 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:42.914135+0000 mgr.y (mgr.15066) 26 : cephadm 1 Reconfiguring iscsi.foo.smithi129.vbvpri (dependencies changed)... 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:42.929361+0000 mgr.y (mgr.15066) 27 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:51.152326+0000 mgr.y (mgr.15066) 36 : cephadm 1 Upgrade: Updating mgr.x 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:11:51.159128+0000 mgr.y (mgr.15066) 37 : cephadm 1 Deploying daemon mgr.x on smithi129 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:16.269542+0000 mgr.y (mgr.15066) 55 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:16.373930+0000 mgr.y (mgr.15066) 58 : cephadm 1 Upgrade: Setting container_image for all mgr 2023-07-29T17:13:35.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:16.410372+0000 mgr.y (mgr.15066) 59 : cephadm 1 Upgrade: It appears safe to stop mon.a 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:18.630942+0000 mgr.y (mgr.15066) 61 : cephadm 1 Upgrade: Updating mon.a 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:18.648913+0000 mgr.y (mgr.15066) 62 : cephadm 1 Deploying daemon mon.a on smithi078 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:33.139239+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:33.225936+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:33.352681+0000 mon.a (mon.0) 38 : cluster 0 Standby manager daemon x restarted 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:33.352894+0000 mon.a (mon.0) 39 : cluster 0 Standby manager daemon x started 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:33.691763+0000 mgr.y (mgr.25169) 1 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Bus STARTING 2023-07-29T17:13:35.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:33.818280+0000 mgr.y (mgr.25169) 2 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Serving on https://172.21.15.78:7150 2023-07-29T17:13:35.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:33.851254+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-29T17:13:35.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:33.920427+0000 mgr.y (mgr.25169) 3 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Serving on http://172.21.15.78:8765 2023-07-29T17:13:35.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:33.921157+0000 mgr.y (mgr.25169) 4 : cephadm 1 [29/Jul/2023:17:12:33] ENGINE Bus STARTED 2023-07-29T17:13:35.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:34.140406+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e30: y(active, since 1.02058s), standbys: x 2023-07-29T17:13:35.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:34.155361+0000 mgr.y (mgr.25169) 5 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:35.132649+0000 mon.a (mon.0) 41 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:35.132713+0000 mon.a (mon.0) 42 : cluster 1 Cluster is now healthy 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:35.155397+0000 mgr.y (mgr.25169) 6 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:36.151205+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e31: y(active, since 3s), standbys: x 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:37.156085+0000 mgr.y (mgr.25169) 10 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:37.270148+0000 mgr.y (mgr.25169) 11 : audit 0 from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:37.843909+0000 mon.b (mon.1) 104 : audit 1 from='client.? 172.21.15.129:0/158598556' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:39.156986+0000 mgr.y (mgr.25169) 12 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:13:35.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:40.182021+0000 mon.a (mon.0) 44 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:40.192107+0000 mon.a (mon.0) 45 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:40.893782+0000 mon.a (mon.0) 46 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:40.903310+0000 mon.a (mon.0) 47 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:40.990950+0000 mon.a (mon.0) 48 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:41.002809+0000 mon.a (mon.0) 49 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:41.157681+0000 mgr.y (mgr.25169) 13 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2023-07-29T17:13:35.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:41.701420+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:41.712500+0000 mon.a (mon.0) 51 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:43.158401+0000 mgr.y (mgr.25169) 14 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2023-07-29T17:13:35.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:45.160084+0000 mgr.y (mgr.25169) 15 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s 2023-07-29T17:13:35.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:47.160880+0000 mgr.y (mgr.25169) 16 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:34.155361+0000 mgr.y (mgr.25169) 5 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:35.132649+0000 mon.a (mon.0) 41 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:35.132713+0000 mon.a (mon.0) 42 : cluster 1 Cluster is now healthy 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:35.155397+0000 mgr.y (mgr.25169) 6 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:36.151205+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e31: y(active, since 3s), standbys: x 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:37.156085+0000 mgr.y (mgr.25169) 10 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:37.270148+0000 mgr.y (mgr.25169) 11 : audit 0 from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:37.843909+0000 mon.b (mon.1) 104 : audit 1 from='client.? 172.21.15.129:0/158598556' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:39.156986+0000 mgr.y (mgr.25169) 12 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:13:35.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:40.182021+0000 mon.a (mon.0) 44 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:40.192107+0000 mon.a (mon.0) 45 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:40.893782+0000 mon.a (mon.0) 46 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:40.903310+0000 mon.a (mon.0) 47 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:40.990950+0000 mon.a (mon.0) 48 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:41.002809+0000 mon.a (mon.0) 49 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:41.157681+0000 mgr.y (mgr.25169) 13 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2023-07-29T17:13:35.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:41.701420+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:41.712500+0000 mon.a (mon.0) 51 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:43.158401+0000 mgr.y (mgr.25169) 14 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2023-07-29T17:13:35.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:45.160084+0000 mgr.y (mgr.25169) 15 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s 2023-07-29T17:13:35.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:47.160880+0000 mgr.y (mgr.25169) 16 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:49.162452+0000 mgr.y (mgr.25169) 17 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:51.163203+0000 mgr.y (mgr.25169) 18 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:51.171698+0000 mon.a (mon.0) 52 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:51.181721+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:53.163956+0000 mgr.y (mgr.25169) 19 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:53.373130+0000 mon.b (mon.1) 105 : audit 0 from='client.? 172.21.15.129:0/720231089' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:53.744700+0000 mon.b (mon.1) 106 : audit 1 from='client.? 172.21.15.129:0/2122394065' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.125639+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.135001+0000 mon.a (mon.0) 55 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.137619+0000 mon.a (mon.0) 56 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.139021+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:55.140241+0000 mgr.y (mgr.25169) 20 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:55.141962+0000 mgr.y (mgr.25169) 21 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.147652+0000 mon.a (mon.0) 58 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.234039+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.238749+0000 mon.a (mon.0) 60 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.242123+0000 mon.a (mon.0) 61 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:35.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:55.244043+0000 mon.a (mon.0) 62 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-29T17:13:35.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:55.245469+0000 mgr.y (mgr.25169) 22 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-07-29T17:13:35.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:56.131866+0000 mon.a (mon.0) 63 : cluster 3 Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:13:35.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:49.162452+0000 mgr.y (mgr.25169) 17 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:51.163203+0000 mgr.y (mgr.25169) 18 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:51.171698+0000 mon.a (mon.0) 52 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:51.181721+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:53.163956+0000 mgr.y (mgr.25169) 19 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:53.373130+0000 mon.b (mon.1) 105 : audit 0 from='client.? 172.21.15.129:0/720231089' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:53.744700+0000 mon.b (mon.1) 106 : audit 1 from='client.? 172.21.15.129:0/2122394065' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.125639+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.135001+0000 mon.a (mon.0) 55 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.137619+0000 mon.a (mon.0) 56 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.139021+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:55.140241+0000 mgr.y (mgr.25169) 20 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:55.141962+0000 mgr.y (mgr.25169) 21 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.147652+0000 mon.a (mon.0) 58 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.234039+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.238749+0000 mon.a (mon.0) 60 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.242123+0000 mon.a (mon.0) 61 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:35.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:55.244043+0000 mon.a (mon.0) 62 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-29T17:13:35.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:55.245469+0000 mgr.y (mgr.25169) 22 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-07-29T17:13:35.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:56.131866+0000 mon.a (mon.0) 63 : cluster 3 Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:13:35.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:57.142651+0000 mgr.y (mgr.25169) 23 : cluster 0 pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:57.514548+0000 mgr.y (mgr.25169) 24 : cephadm 1 Upgrade: Updating mon.c 2023-07-29T17:13:35.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:57.524116+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:57.525945+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:57.527053+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:12:57.528224+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:12:57.529886+0000 mgr.y (mgr.25169) 25 : cephadm 1 Deploying daemon mon.c on smithi078 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:12:59.143567+0000 mgr.y (mgr.25169) 26 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:01.144732+0000 mgr.y (mgr.25169) 27 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:02.809530+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2023-07-29T17:13:35.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:08.492313+0000 mgr.y (mgr.25169) 32 : audit 0 from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.815214+0000 mon.a (mon.0) 83 : cluster 1 mon.a calling monitor election 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.816454+0000 mon.b (mon.1) 109 : cluster 1 mon.b calling monitor election 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.819527+0000 mon.a (mon.0) 84 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.835046+0000 mon.a (mon.0) 85 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.835476+0000 mon.a (mon.0) 86 : cluster 0 fsmap 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.835512+0000 mon.a (mon.0) 87 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.835890+0000 mon.a (mon.0) 88 : cluster 0 mgrmap e31: y(active, since 35s), standbys: x 2023-07-29T17:13:35.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.836038+0000 mon.a (mon.0) 89 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T17:13:35.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.839644+0000 mon.a (mon.0) 90 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.839667+0000 mon.a (mon.0) 91 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.839675+0000 mon.a (mon.0) 92 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:08.839682+0000 mon.a (mon.0) 93 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:57.142651+0000 mgr.y (mgr.25169) 23 : cluster 0 pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:57.514548+0000 mgr.y (mgr.25169) 24 : cephadm 1 Upgrade: Updating mon.c 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:57.524116+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:57.525945+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:57.527053+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:12:57.528224+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:12:57.529886+0000 mgr.y (mgr.25169) 25 : cephadm 1 Deploying daemon mon.c on smithi078 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:12:59.143567+0000 mgr.y (mgr.25169) 26 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:01.144732+0000 mgr.y (mgr.25169) 27 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:02.809530+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:08.492313+0000 mgr.y (mgr.25169) 32 : audit 0 from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.815214+0000 mon.a (mon.0) 83 : cluster 1 mon.a calling monitor election 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.816454+0000 mon.b (mon.1) 109 : cluster 1 mon.b calling monitor election 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.819527+0000 mon.a (mon.0) 84 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.835046+0000 mon.a (mon.0) 85 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.835476+0000 mon.a (mon.0) 86 : cluster 0 fsmap 2023-07-29T17:13:35.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.835512+0000 mon.a (mon.0) 87 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.835890+0000 mon.a (mon.0) 88 : cluster 0 mgrmap e31: y(active, since 35s), standbys: x 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.836038+0000 mon.a (mon.0) 89 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.839644+0000 mon.a (mon.0) 90 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.839667+0000 mon.a (mon.0) 91 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.839675+0000 mon.a (mon.0) 92 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:08.839682+0000 mon.a (mon.0) 93 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:08.852283+0000 mgr.y (mgr.25169) 33 : audit 0 from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:09.014463+0000 mon.b (mon.1) 110 : audit 1 from='client.? 172.21.15.129:0/3636614956' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:09.148744+0000 mgr.y (mgr.25169) 34 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:09.315234+0000 mon.a (mon.0) 94 : audit 0 from='client.34254 172.21.15.78:0/4088428933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:09.693269+0000 mgr.y (mgr.25169) 35 : audit 0 from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:10.127408+0000 mon.a (mon.0) 95 : audit 0 from='client.34266 172.21.15.78:0/2581941085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:11.150593+0000 mgr.y (mgr.25169) 36 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:13.151433+0000 mgr.y (mgr.25169) 37 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:14.413473+0000 mon.a (mon.0) 96 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:14.425544+0000 mon.a (mon.0) 97 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:15.153124+0000 mgr.y (mgr.25169) 38 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:15.223048+0000 mon.a (mon.0) 98 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:15.234260+0000 mon.a (mon.0) 99 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:17.153885+0000 mgr.y (mgr.25169) 39 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:19.155285+0000 mgr.y (mgr.25169) 40 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:21.156870+0000 mgr.y (mgr.25169) 41 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:23.157602+0000 mgr.y (mgr.25169) 42 : cluster 0 pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:25.159335+0000 mgr.y (mgr.25169) 43 : cluster 0 pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.486193+0000 mon.a (mon.0) 100 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.497119+0000 mon.a (mon.0) 101 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.481 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:08.852283+0000 mgr.y (mgr.25169) 33 : audit 0 from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:09.014463+0000 mon.b (mon.1) 110 : audit 1 from='client.? 172.21.15.129:0/3636614956' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: access denied 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:09.148744+0000 mgr.y (mgr.25169) 34 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:09.315234+0000 mon.a (mon.0) 94 : audit 0 from='client.34254 172.21.15.78:0/4088428933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:09.693269+0000 mgr.y (mgr.25169) 35 : audit 0 from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:10.127408+0000 mon.a (mon.0) 95 : audit 0 from='client.34266 172.21.15.78:0/2581941085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:11.150593+0000 mgr.y (mgr.25169) 36 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:13.151433+0000 mgr.y (mgr.25169) 37 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:14.413473+0000 mon.a (mon.0) 96 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:14.425544+0000 mon.a (mon.0) 97 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:15.153124+0000 mgr.y (mgr.25169) 38 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:15.223048+0000 mon.a (mon.0) 98 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:15.234260+0000 mon.a (mon.0) 99 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:17.153885+0000 mgr.y (mgr.25169) 39 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:19.155285+0000 mgr.y (mgr.25169) 40 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:21.156870+0000 mgr.y (mgr.25169) 41 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-29T17:13:35.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:23.157602+0000 mgr.y (mgr.25169) 42 : cluster 0 pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:25.159335+0000 mgr.y (mgr.25169) 43 : cluster 0 pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.486193+0000 mon.a (mon.0) 100 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.497119+0000 mon.a (mon.0) 101 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.499021+0000 mon.a (mon.0) 102 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.500542+0000 mon.a (mon.0) 103 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:13:25.501923+0000 mgr.y (mgr.25169) 44 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:25.503192+0000 mgr.y (mgr.25169) 45 : cluster 0 pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.511989+0000 mon.a (mon.0) 104 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.600876+0000 mon.a (mon.0) 105 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.605319+0000 mon.a (mon.0) 106 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.607573+0000 mon.a (mon.0) 107 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:25.608919+0000 mon.a (mon.0) 108 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-29T17:13:35.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:13:25.610067+0000 mgr.y (mgr.25169) 46 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:27.503937+0000 mgr.y (mgr.25169) 47 : cluster 0 pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:13:28.181361+0000 mgr.y (mgr.25169) 48 : cephadm 1 Upgrade: Updating mon.b 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:28.191152+0000 mon.a (mon.0) 109 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:28.192848+0000 mon.a (mon.0) 110 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:28.194395+0000 mon.a (mon.0) 111 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:28.195803+0000 mon.a (mon.0) 112 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cephadm 2023-07-29T17:13:28.197401+0000 mgr.y (mgr.25169) 49 : cephadm 1 Deploying daemon mon.b on smithi129 2023-07-29T17:13:35.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:29.505619+0000 mgr.y (mgr.25169) 50 : cluster 0 pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:33.507127+0000 mgr.y (mgr.25169) 52 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.094676+0000 mon.a (mon.0) 119 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.094705+0000 mon.a (mon.0) 120 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.499021+0000 mon.a (mon.0) 102 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.500542+0000 mon.a (mon.0) 103 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:35.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:13:25.501923+0000 mgr.y (mgr.25169) 44 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:25.503192+0000 mgr.y (mgr.25169) 45 : cluster 0 pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.511989+0000 mon.a (mon.0) 104 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.600876+0000 mon.a (mon.0) 105 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.605319+0000 mon.a (mon.0) 106 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.607573+0000 mon.a (mon.0) 107 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:25.608919+0000 mon.a (mon.0) 108 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-29T17:13:35.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:13:25.610067+0000 mgr.y (mgr.25169) 46 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:27.503937+0000 mgr.y (mgr.25169) 47 : cluster 0 pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:13:28.181361+0000 mgr.y (mgr.25169) 48 : cephadm 1 Upgrade: Updating mon.b 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:28.191152+0000 mon.a (mon.0) 109 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:28.192848+0000 mon.a (mon.0) 110 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:28.194395+0000 mon.a (mon.0) 111 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:28.195803+0000 mon.a (mon.0) 112 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cephadm 2023-07-29T17:13:28.197401+0000 mgr.y (mgr.25169) 49 : cephadm 1 Deploying daemon mon.b on smithi129 2023-07-29T17:13:35.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:29.505619+0000 mgr.y (mgr.25169) 50 : cluster 0 pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:33.507127+0000 mgr.y (mgr.25169) 52 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:35.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.094676+0000 mon.a (mon.0) 119 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.094705+0000 mon.a (mon.0) 120 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.094718+0000 mon.a (mon.0) 121 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.094733+0000 mon.a (mon.0) 122 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:34.095259+0000 mon.a (mon.0) 123 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.095780+0000 mon.a (mon.0) 124 : cluster 1 mon.a calling monitor election 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.095971+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:34.097385+0000 mon.a (mon.0) 125 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:34.097611+0000 mon.a (mon.0) 126 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.098256+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.099483+0000 mon.a (mon.0) 127 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:35.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.107867+0000 mon.a (mon.0) 128 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.108830+0000 mon.a (mon.0) 129 : cluster 0 fsmap 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.108896+0000 mon.a (mon.0) 130 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.109554+0000 mon.a (mon.0) 131 : cluster 0 mgrmap e31: y(active, since 60s), standbys: x 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.114353+0000 mon.a (mon.0) 132 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.114388+0000 mon.a (mon.0) 133 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.114402+0000 mon.a (mon.0) 134 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: cluster 2023-07-29T17:13:34.114433+0000 mon.a (mon.0) 135 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:34.117832+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[58614]: audit 2023-07-29T17:13:34.126071+0000 mon.a (mon.0) 137 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.094718+0000 mon.a (mon.0) 121 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.094733+0000 mon.a (mon.0) 122 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:34.095259+0000 mon.a (mon.0) 123 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T17:13:35.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.095780+0000 mon.a (mon.0) 124 : cluster 1 mon.a calling monitor election 2023-07-29T17:13:35.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.095971+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:34.097385+0000 mon.a (mon.0) 125 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:34.097611+0000 mon.a (mon.0) 126 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.098256+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.099483+0000 mon.a (mon.0) 127 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.107867+0000 mon.a (mon.0) 128 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.108830+0000 mon.a (mon.0) 129 : cluster 0 fsmap 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.108896+0000 mon.a (mon.0) 130 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2023-07-29T17:13:35.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.109554+0000 mon.a (mon.0) 131 : cluster 0 mgrmap e31: y(active, since 60s), standbys: x 2023-07-29T17:13:35.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.114353+0000 mon.a (mon.0) 132 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:35.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.114388+0000 mon.a (mon.0) 133 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:35.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.114402+0000 mon.a (mon.0) 134 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:35.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: cluster 2023-07-29T17:13:34.114433+0000 mon.a (mon.0) 135 : cluster 3 daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:35.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:34.117832+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:35.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:35 smithi078 bash[60360]: audit 2023-07-29T17:13:34.126071+0000 mon.a (mon.0) 137 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:37.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:37 smithi129 bash[53756]: cluster 2023-07-29T17:13:35.508928+0000 mgr.y (mgr.25169) 53 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:37.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:37 smithi078 bash[58614]: cluster 2023-07-29T17:13:35.508928+0000 mgr.y (mgr.25169) 53 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:37.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:37 smithi078 bash[60360]: cluster 2023-07-29T17:13:35.508928+0000 mgr.y (mgr.25169) 53 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:38.690 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:13:38 smithi078 bash[55873]: debug 2023-07-29T17:13:38.305+0000 7f900e65c700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-29T17:13:39.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:39 smithi129 bash[53756]: cluster 2023-07-29T17:13:37.509679+0000 mgr.y (mgr.25169) 54 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:39.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:39 smithi078 bash[58614]: cluster 2023-07-29T17:13:37.509679+0000 mgr.y (mgr.25169) 54 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:39.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:39 smithi078 bash[60360]: cluster 2023-07-29T17:13:37.509679+0000 mgr.y (mgr.25169) 54 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:40.544 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:13:41.283 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:13:41.284 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3m) 26s ago 9m 10.2M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:13:41.284 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (2m) 0s ago 9m 26.1M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:13:41.284 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 0s ago 7m - - 2023-07-29T17:13:41.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (105s) 0s ago 14m 428M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:13:41.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (2m) 26s ago 15m 464M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (78s) 26s ago 16m 46.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (9s) 0s ago 14m 37.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (39s) 26s ago 14m 35.5M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3m) 26s ago 9m 9148k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (3m) 0s ago 10m 3644k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (13m) 26s ago 13m 55.9M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:13:41.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (13m) 26s ago 13m 53.4M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (12m) 26s ago 12m 48.3M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (12m) 26s ago 12m 53.9M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (12m) 0s ago 12m 55.5M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (11m) 0s ago 11m 53.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (11m) 0s ago 11m 53.3M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (10m) 0s ago 10m 56.5M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:13:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 0s ago 10m - - 2023-07-29T17:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (7m) 26s ago 7m 54.3M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (7m) 0s ago 7m 54.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (7m) 26s ago 7m 54.7M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (7m) 0s ago 7m 54.5M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:13:41.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:41 smithi129 bash[53756]: cluster 2023-07-29T17:13:39.511310+0000 mgr.y (mgr.25169) 55 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:41.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:41 smithi129 bash[53756]: audit 2023-07-29T17:13:40.771340+0000 mon.a (mon.0) 138 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:41.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:41 smithi129 bash[53756]: audit 2023-07-29T17:13:40.782235+0000 mon.a (mon.0) 139 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:41.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:41 smithi078 bash[60360]: cluster 2023-07-29T17:13:39.511310+0000 mgr.y (mgr.25169) 55 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:41.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:41 smithi078 bash[60360]: audit 2023-07-29T17:13:40.771340+0000 mon.a (mon.0) 138 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:41.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:41 smithi078 bash[60360]: audit 2023-07-29T17:13:40.782235+0000 mon.a (mon.0) 139 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:41.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:41 smithi078 bash[58614]: cluster 2023-07-29T17:13:39.511310+0000 mgr.y (mgr.25169) 55 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:41.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:41 smithi078 bash[58614]: audit 2023-07-29T17:13:40.771340+0000 mon.a (mon.0) 138 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:41.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:41 smithi078 bash[58614]: audit 2023-07-29T17:13:40.782235+0000 mon.a (mon.0) 139 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:41.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:13:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:13:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:13:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:13:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:13:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:13:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:13:41.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2023-07-29T17:13:41.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 5 2023-07-29T17:13:41.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:13:41.733 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:13:42.106 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:13:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:13:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:13:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:13:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:13:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:13:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:13:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:13:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "5/23 daemons upgraded", 2023-07-29T17:13:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2023-07-29T17:13:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:13:42.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:42 smithi129 bash[53756]: audit 2023-07-29T17:13:40.510715+0000 mgr.y (mgr.25169) 56 : audit 0 from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:42 smithi129 bash[53756]: audit 2023-07-29T17:13:40.887022+0000 mgr.y (mgr.25169) 57 : audit 0 from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:42 smithi129 bash[53756]: audit 2023-07-29T17:13:41.587711+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:42 smithi129 bash[53756]: audit 2023-07-29T17:13:41.597752+0000 mon.a (mon.0) 141 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:42 smithi129 bash[53756]: audit 2023-07-29T17:13:41.711048+0000 mon.c (mon.2) 3 : audit 0 from='client.44120 172.21.15.78:0/890160400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:42.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[58614]: audit 2023-07-29T17:13:40.510715+0000 mgr.y (mgr.25169) 56 : audit 0 from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:42.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[58614]: audit 2023-07-29T17:13:40.887022+0000 mgr.y (mgr.25169) 57 : audit 0 from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:42.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[58614]: audit 2023-07-29T17:13:41.587711+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:42.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[58614]: audit 2023-07-29T17:13:41.597752+0000 mon.a (mon.0) 141 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:42.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[58614]: audit 2023-07-29T17:13:41.711048+0000 mon.c (mon.2) 3 : audit 0 from='client.44120 172.21.15.78:0/890160400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:42.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[60360]: audit 2023-07-29T17:13:40.510715+0000 mgr.y (mgr.25169) 56 : audit 0 from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:42.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[60360]: audit 2023-07-29T17:13:40.887022+0000 mgr.y (mgr.25169) 57 : audit 0 from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:42.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[60360]: audit 2023-07-29T17:13:41.587711+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:42.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[60360]: audit 2023-07-29T17:13:41.597752+0000 mon.a (mon.0) 141 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:42.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:42 smithi078 bash[60360]: audit 2023-07-29T17:13:41.711048+0000 mon.c (mon.2) 3 : audit 0 from='client.44120 172.21.15.78:0/890160400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:13:42.548 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:13:42.548 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:13:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:13:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:13:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:43 smithi129 bash[53756]: audit 2023-07-29T17:13:41.248622+0000 mgr.y (mgr.25169) 58 : audit 0 from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:43 smithi129 bash[53756]: cluster 2023-07-29T17:13:41.512047+0000 mgr.y (mgr.25169) 59 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:43 smithi129 bash[53756]: audit 2023-07-29T17:13:42.086690+0000 mgr.y (mgr.25169) 60 : audit 0 from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:43 smithi129 bash[53756]: audit 2023-07-29T17:13:42.528973+0000 mon.a (mon.0) 142 : audit 0 from='client.34302 172.21.15.78:0/1336587987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:43.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[58614]: audit 2023-07-29T17:13:41.248622+0000 mgr.y (mgr.25169) 58 : audit 0 from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:43.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[58614]: cluster 2023-07-29T17:13:41.512047+0000 mgr.y (mgr.25169) 59 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:43.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[58614]: audit 2023-07-29T17:13:42.086690+0000 mgr.y (mgr.25169) 60 : audit 0 from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:43.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[58614]: audit 2023-07-29T17:13:42.528973+0000 mon.a (mon.0) 142 : audit 0 from='client.34302 172.21.15.78:0/1336587987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:43.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[60360]: audit 2023-07-29T17:13:41.248622+0000 mgr.y (mgr.25169) 58 : audit 0 from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:43.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[60360]: cluster 2023-07-29T17:13:41.512047+0000 mgr.y (mgr.25169) 59 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:43.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[60360]: audit 2023-07-29T17:13:42.086690+0000 mgr.y (mgr.25169) 60 : audit 0 from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:13:43.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:43 smithi078 bash[60360]: audit 2023-07-29T17:13:42.528973+0000 mon.a (mon.0) 142 : audit 0 from='client.34302 172.21.15.78:0/1336587987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:13:45.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:45 smithi129 bash[53756]: cluster 2023-07-29T17:13:43.512825+0000 mgr.y (mgr.25169) 61 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:45.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:45 smithi078 bash[58614]: cluster 2023-07-29T17:13:43.512825+0000 mgr.y (mgr.25169) 61 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:45.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:45 smithi078 bash[60360]: cluster 2023-07-29T17:13:43.512825+0000 mgr.y (mgr.25169) 61 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:47.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:47 smithi129 bash[53756]: cluster 2023-07-29T17:13:45.514510+0000 mgr.y (mgr.25169) 62 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:47.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:47 smithi078 bash[60360]: cluster 2023-07-29T17:13:45.514510+0000 mgr.y (mgr.25169) 62 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:47.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:47 smithi078 bash[58614]: cluster 2023-07-29T17:13:45.514510+0000 mgr.y (mgr.25169) 62 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:49.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:49 smithi129 bash[53756]: cluster 2023-07-29T17:13:47.515323+0000 mgr.y (mgr.25169) 63 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:49.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:49 smithi078 bash[58614]: cluster 2023-07-29T17:13:47.515323+0000 mgr.y (mgr.25169) 63 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:49.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:49 smithi078 bash[60360]: cluster 2023-07-29T17:13:47.515323+0000 mgr.y (mgr.25169) 63 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:51.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:51 smithi129 bash[53756]: cluster 2023-07-29T17:13:49.516945+0000 mgr.y (mgr.25169) 64 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:51.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:51 smithi078 bash[58614]: cluster 2023-07-29T17:13:49.516945+0000 mgr.y (mgr.25169) 64 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:51.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:51 smithi078 bash[60360]: cluster 2023-07-29T17:13:49.516945+0000 mgr.y (mgr.25169) 64 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: cluster 2023-07-29T17:13:51.517688+0000 mgr.y (mgr.25169) 65 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.002900+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.012105+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.013760+0000 mon.a (mon.0) 145 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.015344+0000 mon.a (mon.0) 146 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: cephadm 2023-07-29T17:13:52.016734+0000 mgr.y (mgr.25169) 66 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: cluster 2023-07-29T17:13:52.017781+0000 mgr.y (mgr.25169) 67 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: cluster 2023-07-29T17:13:52.018263+0000 mgr.y (mgr.25169) 68 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.025401+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: cephadm 2023-07-29T17:13:52.099083+0000 mgr.y (mgr.25169) 69 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-07-29T17:13:53.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.099802+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:53.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.101608+0000 mon.a (mon.0) 149 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:53.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: audit 2023-07-29T17:13:52.103433+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:53.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:53 smithi129 bash[53756]: cephadm 2023-07-29T17:13:52.105366+0000 mgr.y (mgr.25169) 70 : cephadm 1 Reconfiguring daemon mon.a on smithi078 2023-07-29T17:13:53.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: cluster 2023-07-29T17:13:51.517688+0000 mgr.y (mgr.25169) 65 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.002900+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.012105+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.013760+0000 mon.a (mon.0) 145 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:53.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.015344+0000 mon.a (mon.0) 146 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: cephadm 2023-07-29T17:13:52.016734+0000 mgr.y (mgr.25169) 66 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: cluster 2023-07-29T17:13:52.017781+0000 mgr.y (mgr.25169) 67 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: cluster 2023-07-29T17:13:52.018263+0000 mgr.y (mgr.25169) 68 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.025401+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: cephadm 2023-07-29T17:13:52.099083+0000 mgr.y (mgr.25169) 69 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.099802+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.101608+0000 mon.a (mon.0) 149 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:53.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: audit 2023-07-29T17:13:52.103433+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:53.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[58614]: cephadm 2023-07-29T17:13:52.105366+0000 mgr.y (mgr.25169) 70 : cephadm 1 Reconfiguring daemon mon.a on smithi078 2023-07-29T17:13:53.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: cluster 2023-07-29T17:13:51.517688+0000 mgr.y (mgr.25169) 65 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.002900+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.012105+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.013760+0000 mon.a (mon.0) 145 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:53.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.015344+0000 mon.a (mon.0) 146 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:13:53.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: cephadm 2023-07-29T17:13:52.016734+0000 mgr.y (mgr.25169) 66 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:13:53.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: cluster 2023-07-29T17:13:52.017781+0000 mgr.y (mgr.25169) 67 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: cluster 2023-07-29T17:13:52.018263+0000 mgr.y (mgr.25169) 68 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:53.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.025401+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:53.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: cephadm 2023-07-29T17:13:52.099083+0000 mgr.y (mgr.25169) 69 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-07-29T17:13:53.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.099802+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:53.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.101608+0000 mon.a (mon.0) 149 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:53.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: audit 2023-07-29T17:13:52.103433+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:53.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:53 smithi078 bash[60360]: cephadm 2023-07-29T17:13:52.105366+0000 mgr.y (mgr.25169) 70 : cephadm 1 Reconfiguring daemon mon.a on smithi078 2023-07-29T17:13:54.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[60360]: cluster 2023-07-29T17:13:54.019019+0000 mgr.y (mgr.25169) 71 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:54.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[60360]: audit 2023-07-29T17:13:54.373228+0000 mon.a (mon.0) 151 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:54.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[60360]: audit 2023-07-29T17:13:54.383361+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:54.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[60360]: audit 2023-07-29T17:13:54.426517+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:13:54.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[60360]: audit 2023-07-29T17:13:54.428576+0000 mon.a (mon.0) 154 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:13:54.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[60360]: audit 2023-07-29T17:13:54.430559+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:54.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[58614]: cluster 2023-07-29T17:13:54.019019+0000 mgr.y (mgr.25169) 71 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:54.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[58614]: audit 2023-07-29T17:13:54.373228+0000 mon.a (mon.0) 151 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:54.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[58614]: audit 2023-07-29T17:13:54.383361+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:54.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[58614]: audit 2023-07-29T17:13:54.426517+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:13:54.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[58614]: audit 2023-07-29T17:13:54.428576+0000 mon.a (mon.0) 154 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:13:54.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:54 smithi078 bash[58614]: audit 2023-07-29T17:13:54.430559+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:55.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:54 smithi129 bash[53756]: cluster 2023-07-29T17:13:54.019019+0000 mgr.y (mgr.25169) 71 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:55.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:54 smithi129 bash[53756]: audit 2023-07-29T17:13:54.373228+0000 mon.a (mon.0) 151 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:55.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:54 smithi129 bash[53756]: audit 2023-07-29T17:13:54.383361+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:55.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:54 smithi129 bash[53756]: audit 2023-07-29T17:13:54.426517+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:13:55.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:54 smithi129 bash[53756]: audit 2023-07-29T17:13:54.428576+0000 mon.a (mon.0) 154 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:13:55.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:54 smithi129 bash[53756]: audit 2023-07-29T17:13:54.430559+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:55.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:55 smithi078 bash[58614]: cephadm 2023-07-29T17:13:54.425796+0000 mgr.y (mgr.25169) 72 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-07-29T17:13:55.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:55 smithi078 bash[58614]: cephadm 2023-07-29T17:13:54.432645+0000 mgr.y (mgr.25169) 73 : cephadm 1 Reconfiguring daemon mgr.y on smithi078 2023-07-29T17:13:55.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:55 smithi078 bash[60360]: cephadm 2023-07-29T17:13:54.425796+0000 mgr.y (mgr.25169) 72 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-07-29T17:13:55.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:55 smithi078 bash[60360]: cephadm 2023-07-29T17:13:54.432645+0000 mgr.y (mgr.25169) 73 : cephadm 1 Reconfiguring daemon mgr.y on smithi078 2023-07-29T17:13:56.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:55 smithi129 bash[53756]: cephadm 2023-07-29T17:13:54.425796+0000 mgr.y (mgr.25169) 72 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-07-29T17:13:56.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:55 smithi129 bash[53756]: cephadm 2023-07-29T17:13:54.432645+0000 mgr.y (mgr.25169) 73 : cephadm 1 Reconfiguring daemon mgr.y on smithi078 2023-07-29T17:13:56.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:56 smithi078 bash[58614]: cluster 2023-07-29T17:13:56.020711+0000 mgr.y (mgr.25169) 74 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:56.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:56 smithi078 bash[60360]: cluster 2023-07-29T17:13:56.020711+0000 mgr.y (mgr.25169) 74 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:57.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:56 smithi129 bash[53756]: cluster 2023-07-29T17:13:56.020711+0000 mgr.y (mgr.25169) 74 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:57.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: audit 2023-07-29T17:13:56.666025+0000 mon.a (mon.0) 156 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:57.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: audit 2023-07-29T17:13:56.679154+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:57.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: cephadm 2023-07-29T17:13:56.701605+0000 mgr.y (mgr.25169) 75 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-07-29T17:13:57.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: audit 2023-07-29T17:13:56.702200+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:57.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: audit 2023-07-29T17:13:56.704090+0000 mon.a (mon.0) 159 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:57.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: audit 2023-07-29T17:13:56.705808+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:57.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[58614]: cephadm 2023-07-29T17:13:56.707902+0000 mgr.y (mgr.25169) 76 : cephadm 1 Reconfiguring daemon mon.c on smithi078 2023-07-29T17:13:57.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: audit 2023-07-29T17:13:56.666025+0000 mon.a (mon.0) 156 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:57.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: audit 2023-07-29T17:13:56.679154+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:57.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: cephadm 2023-07-29T17:13:56.701605+0000 mgr.y (mgr.25169) 75 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-07-29T17:13:57.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: audit 2023-07-29T17:13:56.702200+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:57.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: audit 2023-07-29T17:13:56.704090+0000 mon.a (mon.0) 159 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:57.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: audit 2023-07-29T17:13:56.705808+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:57.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:57 smithi078 bash[60360]: cephadm 2023-07-29T17:13:56.707902+0000 mgr.y (mgr.25169) 76 : cephadm 1 Reconfiguring daemon mon.c on smithi078 2023-07-29T17:13:58.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: audit 2023-07-29T17:13:56.666025+0000 mon.a (mon.0) 156 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:58.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: audit 2023-07-29T17:13:56.679154+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:13:58.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: cephadm 2023-07-29T17:13:56.701605+0000 mgr.y (mgr.25169) 75 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-07-29T17:13:58.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: audit 2023-07-29T17:13:56.702200+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:13:58.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: audit 2023-07-29T17:13:56.704090+0000 mon.a (mon.0) 159 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:13:58.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: audit 2023-07-29T17:13:56.705808+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:13:58.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:57 smithi129 bash[53756]: cephadm 2023-07-29T17:13:56.707902+0000 mgr.y (mgr.25169) 76 : cephadm 1 Reconfiguring daemon mon.c on smithi078 2023-07-29T17:13:59.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:58 smithi129 bash[53756]: cluster 2023-07-29T17:13:58.021244+0000 mgr.y (mgr.25169) 77 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:59.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:58 smithi078 bash[58614]: cluster 2023-07-29T17:13:58.021244+0000 mgr.y (mgr.25169) 77 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:13:59.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:58 smithi078 bash[60360]: cluster 2023-07-29T17:13:58.021244+0000 mgr.y (mgr.25169) 77 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:00.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:59 smithi129 bash[53756]: audit 2023-07-29T17:13:58.857147+0000 mon.a (mon.0) 161 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:00.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:59 smithi129 bash[53756]: audit 2023-07-29T17:13:58.865785+0000 mon.a (mon.0) 162 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:00.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:59 smithi129 bash[53756]: cephadm 2023-07-29T17:13:58.877552+0000 mgr.y (mgr.25169) 78 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-07-29T17:14:00.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:59 smithi129 bash[53756]: audit 2023-07-29T17:13:58.878160+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T17:14:00.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:59 smithi129 bash[53756]: audit 2023-07-29T17:13:58.879856+0000 mon.a (mon.0) 164 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:00.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:13:59 smithi129 bash[53756]: cephadm 2023-07-29T17:13:58.885030+0000 mgr.y (mgr.25169) 79 : cephadm 1 Reconfiguring daemon osd.0 on smithi078 2023-07-29T17:14:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[58614]: audit 2023-07-29T17:13:58.857147+0000 mon.a (mon.0) 161 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[58614]: audit 2023-07-29T17:13:58.865785+0000 mon.a (mon.0) 162 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[58614]: cephadm 2023-07-29T17:13:58.877552+0000 mgr.y (mgr.25169) 78 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-07-29T17:14:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[58614]: audit 2023-07-29T17:13:58.878160+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T17:14:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[58614]: audit 2023-07-29T17:13:58.879856+0000 mon.a (mon.0) 164 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[58614]: cephadm 2023-07-29T17:13:58.885030+0000 mgr.y (mgr.25169) 79 : cephadm 1 Reconfiguring daemon osd.0 on smithi078 2023-07-29T17:14:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[60360]: audit 2023-07-29T17:13:58.857147+0000 mon.a (mon.0) 161 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[60360]: audit 2023-07-29T17:13:58.865785+0000 mon.a (mon.0) 162 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[60360]: cephadm 2023-07-29T17:13:58.877552+0000 mgr.y (mgr.25169) 78 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-07-29T17:14:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[60360]: audit 2023-07-29T17:13:58.878160+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T17:14:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[60360]: audit 2023-07-29T17:13:58.879856+0000 mon.a (mon.0) 164 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:13:59 smithi078 bash[60360]: cephadm 2023-07-29T17:13:58.885030+0000 mgr.y (mgr.25169) 79 : cephadm 1 Reconfiguring daemon osd.0 on smithi078 2023-07-29T17:14:01.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:00 smithi129 bash[53756]: cluster 2023-07-29T17:14:00.022869+0000 mgr.y (mgr.25169) 80 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:01.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:00 smithi078 bash[60360]: cluster 2023-07-29T17:14:00.022869+0000 mgr.y (mgr.25169) 80 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:01.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:00 smithi078 bash[58614]: cluster 2023-07-29T17:14:00.022869+0000 mgr.y (mgr.25169) 80 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:02.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: audit 2023-07-29T17:14:01.285457+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:02.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: audit 2023-07-29T17:14:01.293909+0000 mon.a (mon.0) 166 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:02.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: cephadm 2023-07-29T17:14:01.304313+0000 mgr.y (mgr.25169) 81 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-07-29T17:14:02.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: audit 2023-07-29T17:14:01.304807+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:14:02.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: audit 2023-07-29T17:14:01.306082+0000 mon.a (mon.0) 168 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:02.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: cephadm 2023-07-29T17:14:01.309463+0000 mgr.y (mgr.25169) 82 : cephadm 1 Reconfiguring daemon osd.1 on smithi078 2023-07-29T17:14:02.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[58614]: cluster 2023-07-29T17:14:02.023594+0000 mgr.y (mgr.25169) 83 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:02.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: audit 2023-07-29T17:14:01.285457+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:02.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: audit 2023-07-29T17:14:01.293909+0000 mon.a (mon.0) 166 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:02.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: cephadm 2023-07-29T17:14:01.304313+0000 mgr.y (mgr.25169) 81 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-07-29T17:14:02.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: audit 2023-07-29T17:14:01.304807+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:14:02.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: audit 2023-07-29T17:14:01.306082+0000 mon.a (mon.0) 168 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:02.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: cephadm 2023-07-29T17:14:01.309463+0000 mgr.y (mgr.25169) 82 : cephadm 1 Reconfiguring daemon osd.1 on smithi078 2023-07-29T17:14:02.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:02 smithi078 bash[60360]: cluster 2023-07-29T17:14:02.023594+0000 mgr.y (mgr.25169) 83 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:02.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: audit 2023-07-29T17:14:01.285457+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:02.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: audit 2023-07-29T17:14:01.293909+0000 mon.a (mon.0) 166 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:02.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: cephadm 2023-07-29T17:14:01.304313+0000 mgr.y (mgr.25169) 81 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-07-29T17:14:02.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: audit 2023-07-29T17:14:01.304807+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:14:02.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: audit 2023-07-29T17:14:01.306082+0000 mon.a (mon.0) 168 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:02.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: cephadm 2023-07-29T17:14:01.309463+0000 mgr.y (mgr.25169) 82 : cephadm 1 Reconfiguring daemon osd.1 on smithi078 2023-07-29T17:14:02.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:02 smithi129 bash[53756]: cluster 2023-07-29T17:14:02.023594+0000 mgr.y (mgr.25169) 83 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: audit 2023-07-29T17:14:03.517828+0000 mon.a (mon.0) 169 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: audit 2023-07-29T17:14:03.526867+0000 mon.a (mon.0) 170 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: cephadm 2023-07-29T17:14:03.577435+0000 mgr.y (mgr.25169) 84 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-07-29T17:14:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: audit 2023-07-29T17:14:03.578123+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:14:04.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: audit 2023-07-29T17:14:03.579963+0000 mon.a (mon.0) 172 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:04.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: cephadm 2023-07-29T17:14:03.585591+0000 mgr.y (mgr.25169) 85 : cephadm 1 Reconfiguring daemon osd.2 on smithi078 2023-07-29T17:14:04.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[58614]: cluster 2023-07-29T17:14:04.024397+0000 mgr.y (mgr.25169) 86 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:04.842 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: audit 2023-07-29T17:14:03.517828+0000 mon.a (mon.0) 169 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:04.842 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: audit 2023-07-29T17:14:03.526867+0000 mon.a (mon.0) 170 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:04.842 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: cephadm 2023-07-29T17:14:03.577435+0000 mgr.y (mgr.25169) 84 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-07-29T17:14:04.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: audit 2023-07-29T17:14:03.578123+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:14:04.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: audit 2023-07-29T17:14:03.579963+0000 mon.a (mon.0) 172 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:04.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: cephadm 2023-07-29T17:14:03.585591+0000 mgr.y (mgr.25169) 85 : cephadm 1 Reconfiguring daemon osd.2 on smithi078 2023-07-29T17:14:04.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:04 smithi078 bash[60360]: cluster 2023-07-29T17:14:04.024397+0000 mgr.y (mgr.25169) 86 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:04.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: audit 2023-07-29T17:14:03.517828+0000 mon.a (mon.0) 169 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:04.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: audit 2023-07-29T17:14:03.526867+0000 mon.a (mon.0) 170 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:04.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: cephadm 2023-07-29T17:14:03.577435+0000 mgr.y (mgr.25169) 84 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-07-29T17:14:04.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: audit 2023-07-29T17:14:03.578123+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:14:04.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: audit 2023-07-29T17:14:03.579963+0000 mon.a (mon.0) 172 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:04.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: cephadm 2023-07-29T17:14:03.585591+0000 mgr.y (mgr.25169) 85 : cephadm 1 Reconfiguring daemon osd.2 on smithi078 2023-07-29T17:14:04.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:04 smithi129 bash[53756]: cluster 2023-07-29T17:14:04.024397+0000 mgr.y (mgr.25169) 86 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:07.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: audit 2023-07-29T17:14:05.731296+0000 mon.a (mon.0) 173 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:07.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: audit 2023-07-29T17:14:05.738760+0000 mon.a (mon.0) 174 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:07.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: cephadm 2023-07-29T17:14:05.753860+0000 mgr.y (mgr.25169) 87 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-07-29T17:14:07.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: audit 2023-07-29T17:14:05.754365+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:14:07.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: audit 2023-07-29T17:14:05.755815+0000 mon.a (mon.0) 176 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: cephadm 2023-07-29T17:14:05.760007+0000 mgr.y (mgr.25169) 88 : cephadm 1 Reconfiguring daemon osd.3 on smithi078 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[60360]: cluster 2023-07-29T17:14:06.026067+0000 mgr.y (mgr.25169) 89 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: audit 2023-07-29T17:14:05.731296+0000 mon.a (mon.0) 173 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: audit 2023-07-29T17:14:05.738760+0000 mon.a (mon.0) 174 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: cephadm 2023-07-29T17:14:05.753860+0000 mgr.y (mgr.25169) 87 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: audit 2023-07-29T17:14:05.754365+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:14:07.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: audit 2023-07-29T17:14:05.755815+0000 mon.a (mon.0) 176 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:07.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: cephadm 2023-07-29T17:14:05.760007+0000 mgr.y (mgr.25169) 88 : cephadm 1 Reconfiguring daemon osd.3 on smithi078 2023-07-29T17:14:07.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:06 smithi078 bash[58614]: cluster 2023-07-29T17:14:06.026067+0000 mgr.y (mgr.25169) 89 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:07.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: audit 2023-07-29T17:14:05.731296+0000 mon.a (mon.0) 173 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:07.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: audit 2023-07-29T17:14:05.738760+0000 mon.a (mon.0) 174 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:07.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: cephadm 2023-07-29T17:14:05.753860+0000 mgr.y (mgr.25169) 87 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-07-29T17:14:07.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: audit 2023-07-29T17:14:05.754365+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:14:07.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: audit 2023-07-29T17:14:05.755815+0000 mon.a (mon.0) 176 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:07.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: cephadm 2023-07-29T17:14:05.760007+0000 mgr.y (mgr.25169) 88 : cephadm 1 Reconfiguring daemon osd.3 on smithi078 2023-07-29T17:14:07.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:06 smithi129 bash[53756]: cluster 2023-07-29T17:14:06.026067+0000 mgr.y (mgr.25169) 89 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:09.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: audit 2023-07-29T17:14:07.985783+0000 mon.a (mon.0) 177 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:09.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: audit 2023-07-29T17:14:07.994692+0000 mon.a (mon.0) 178 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:09.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: cluster 2023-07-29T17:14:08.026758+0000 mgr.y (mgr.25169) 90 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:09.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: cephadm 2023-07-29T17:14:08.031156+0000 mgr.y (mgr.25169) 91 : cephadm 1 Reconfiguring rgw.foo.smithi078.lydpyb (monmap changed)... 2023-07-29T17:14:09.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: audit 2023-07-29T17:14:08.032037+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:09.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: audit 2023-07-29T17:14:08.035791+0000 mon.a (mon.0) 180 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:09.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[60360]: cephadm 2023-07-29T17:14:08.037695+0000 mgr.y (mgr.25169) 92 : cephadm 1 Reconfiguring daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:14:09.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: audit 2023-07-29T17:14:07.985783+0000 mon.a (mon.0) 177 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:09.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: audit 2023-07-29T17:14:07.994692+0000 mon.a (mon.0) 178 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:09.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: cluster 2023-07-29T17:14:08.026758+0000 mgr.y (mgr.25169) 90 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:09.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: cephadm 2023-07-29T17:14:08.031156+0000 mgr.y (mgr.25169) 91 : cephadm 1 Reconfiguring rgw.foo.smithi078.lydpyb (monmap changed)... 2023-07-29T17:14:09.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: audit 2023-07-29T17:14:08.032037+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:09.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: audit 2023-07-29T17:14:08.035791+0000 mon.a (mon.0) 180 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:09.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:08 smithi078 bash[58614]: cephadm 2023-07-29T17:14:08.037695+0000 mgr.y (mgr.25169) 92 : cephadm 1 Reconfiguring daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:14:09.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: audit 2023-07-29T17:14:07.985783+0000 mon.a (mon.0) 177 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:09.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: audit 2023-07-29T17:14:07.994692+0000 mon.a (mon.0) 178 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:09.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: cluster 2023-07-29T17:14:08.026758+0000 mgr.y (mgr.25169) 90 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: cephadm 2023-07-29T17:14:08.031156+0000 mgr.y (mgr.25169) 91 : cephadm 1 Reconfiguring rgw.foo.smithi078.lydpyb (monmap changed)... 2023-07-29T17:14:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: audit 2023-07-29T17:14:08.032037+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: audit 2023-07-29T17:14:08.035791+0000 mon.a (mon.0) 180 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:08 smithi129 bash[53756]: cephadm 2023-07-29T17:14:08.037695+0000 mgr.y (mgr.25169) 92 : cephadm 1 Reconfiguring daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:14:11.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[58614]: cluster 2023-07-29T17:14:10.028500+0000 mgr.y (mgr.25169) 93 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:11.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[58614]: audit 2023-07-29T17:14:10.162673+0000 mon.a (mon.0) 181 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:11.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[58614]: audit 2023-07-29T17:14:10.171326+0000 mon.a (mon.0) 182 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:11.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[58614]: audit 2023-07-29T17:14:10.207747+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:11.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[58614]: audit 2023-07-29T17:14:10.212016+0000 mon.a (mon.0) 184 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:11.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[60360]: cluster 2023-07-29T17:14:10.028500+0000 mgr.y (mgr.25169) 93 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:11.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[60360]: audit 2023-07-29T17:14:10.162673+0000 mon.a (mon.0) 181 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:11.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[60360]: audit 2023-07-29T17:14:10.171326+0000 mon.a (mon.0) 182 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:11.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[60360]: audit 2023-07-29T17:14:10.207747+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:11.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:11 smithi078 bash[60360]: audit 2023-07-29T17:14:10.212016+0000 mon.a (mon.0) 184 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:11.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:11 smithi129 bash[53756]: cluster 2023-07-29T17:14:10.028500+0000 mgr.y (mgr.25169) 93 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:11.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:11 smithi129 bash[53756]: audit 2023-07-29T17:14:10.162673+0000 mon.a (mon.0) 181 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:11.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:11 smithi129 bash[53756]: audit 2023-07-29T17:14:10.171326+0000 mon.a (mon.0) 182 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:11.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:11 smithi129 bash[53756]: audit 2023-07-29T17:14:10.207747+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:11.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:11 smithi129 bash[53756]: audit 2023-07-29T17:14:10.212016+0000 mon.a (mon.0) 184 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:12.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:12 smithi078 bash[60360]: cephadm 2023-07-29T17:14:10.206921+0000 mgr.y (mgr.25169) 94 : cephadm 1 Reconfiguring rgw.smpl.smithi078.xbxtjr (monmap changed)... 2023-07-29T17:14:12.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:12 smithi078 bash[60360]: cephadm 2023-07-29T17:14:10.214128+0000 mgr.y (mgr.25169) 95 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:14:12.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:12 smithi078 bash[58614]: cephadm 2023-07-29T17:14:10.206921+0000 mgr.y (mgr.25169) 94 : cephadm 1 Reconfiguring rgw.smpl.smithi078.xbxtjr (monmap changed)... 2023-07-29T17:14:12.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:12 smithi078 bash[58614]: cephadm 2023-07-29T17:14:10.214128+0000 mgr.y (mgr.25169) 95 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:14:12.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:12 smithi129 bash[53756]: cephadm 2023-07-29T17:14:10.206921+0000 mgr.y (mgr.25169) 94 : cephadm 1 Reconfiguring rgw.smpl.smithi078.xbxtjr (monmap changed)... 2023-07-29T17:14:12.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:12 smithi129 bash[53756]: cephadm 2023-07-29T17:14:10.214128+0000 mgr.y (mgr.25169) 95 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:14:12.951 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:14:13.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: cluster 2023-07-29T17:14:12.029308+0000 mgr.y (mgr.25169) 96 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:13.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: audit 2023-07-29T17:14:12.203507+0000 mon.a (mon.0) 185 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:13.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: audit 2023-07-29T17:14:12.214331+0000 mon.a (mon.0) 186 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:13.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: cephadm 2023-07-29T17:14:12.234087+0000 mgr.y (mgr.25169) 97 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-07-29T17:14:13.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: audit 2023-07-29T17:14:12.234783+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:14:13.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: audit 2023-07-29T17:14:12.236546+0000 mon.a (mon.0) 188 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:14:13.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: audit 2023-07-29T17:14:12.238135+0000 mon.a (mon.0) 189 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:13.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: cephadm 2023-07-29T17:14:12.240141+0000 mgr.y (mgr.25169) 98 : cephadm 1 Reconfiguring daemon mon.b on smithi129 2023-07-29T17:14:13.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[60360]: audit 2023-07-29T17:14:12.916691+0000 mgr.y (mgr.25169) 99 : audit 0 from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:13.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: cluster 2023-07-29T17:14:12.029308+0000 mgr.y (mgr.25169) 96 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: audit 2023-07-29T17:14:12.203507+0000 mon.a (mon.0) 185 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: audit 2023-07-29T17:14:12.214331+0000 mon.a (mon.0) 186 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: cephadm 2023-07-29T17:14:12.234087+0000 mgr.y (mgr.25169) 97 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: audit 2023-07-29T17:14:12.234783+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: audit 2023-07-29T17:14:12.236546+0000 mon.a (mon.0) 188 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: audit 2023-07-29T17:14:12.238135+0000 mon.a (mon.0) 189 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: cephadm 2023-07-29T17:14:12.240141+0000 mgr.y (mgr.25169) 98 : cephadm 1 Reconfiguring daemon mon.b on smithi129 2023-07-29T17:14:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:13 smithi078 bash[58614]: audit 2023-07-29T17:14:12.916691+0000 mgr.y (mgr.25169) 99 : audit 0 from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:13.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: cluster 2023-07-29T17:14:12.029308+0000 mgr.y (mgr.25169) 96 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:13.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: audit 2023-07-29T17:14:12.203507+0000 mon.a (mon.0) 185 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: audit 2023-07-29T17:14:12.214331+0000 mon.a (mon.0) 186 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: cephadm 2023-07-29T17:14:12.234087+0000 mgr.y (mgr.25169) 97 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: audit 2023-07-29T17:14:12.234783+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: audit 2023-07-29T17:14:12.236546+0000 mon.a (mon.0) 188 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: audit 2023-07-29T17:14:12.238135+0000 mon.a (mon.0) 189 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: cephadm 2023-07-29T17:14:12.240141+0000 mgr.y (mgr.25169) 98 : cephadm 1 Reconfiguring daemon mon.b on smithi129 2023-07-29T17:14:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:13 smithi129 bash[53756]: audit 2023-07-29T17:14:12.916691+0000 mgr.y (mgr.25169) 99 : audit 0 from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:14:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4m) 59s ago 10m 10.2M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:14:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (3m) 32s ago 10m 26.1M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:14:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 32s ago 8m - - 2023-07-29T17:14:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (2m) 32s ago 14m 428M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:14:13.688 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (3m) 59s ago 16m 464M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:14:13.688 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (111s) 59s ago 16m 46.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:14:13.688 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (41s) 32s ago 15m 37.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:14:13.688 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (72s) 59s ago 15m 35.5M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:14:13.688 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4m) 59s ago 10m 9148k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (3m) 32s ago 10m 3644k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (14m) 59s ago 14m 55.9M 4096M 16.2.0 24ecd6d5f14c c46dd67e398b 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (13m) 59s ago 13m 53.4M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (13m) 59s ago 13m 48.3M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (12m) 59s ago 13m 53.9M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (12m) 32s ago 12m 55.5M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:14:13.689 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (12m) 32s ago 12m 53.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (11m) 32s ago 11m 53.3M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (11m) 32s ago 11m 56.5M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 32s ago 10m - - 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (8m) 59s ago 8m 54.3M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (8m) 32s ago 8m 54.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (8m) 59s ago 8m 54.7M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:14:13.690 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (8m) 32s ago 8m 54.5M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:14:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:14:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:14:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:14:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:14:14.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:14:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:14:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-29T17:14:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:14:14.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:14:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:14:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2023-07-29T17:14:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 5 2023-07-29T17:14:14.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:14:14.128 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:14:14.494 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:14:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:14:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:14:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "5/23 daemons upgraded", 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2023-07-29T17:14:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:14:14.496 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:14:14.934 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:14:14.935 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:14:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:14:14.935 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:14:15.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:14 smithi129 bash[53756]: audit 2023-07-29T17:14:13.290820+0000 mgr.y (mgr.25169) 100 : audit 0 from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:15.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:14 smithi129 bash[53756]: audit 2023-07-29T17:14:13.651802+0000 mgr.y (mgr.25169) 101 : audit 0 from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:15.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:14 smithi129 bash[53756]: cluster 2023-07-29T17:14:14.030136+0000 mgr.y (mgr.25169) 102 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:15.079 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:14 smithi129 bash[53756]: audit 2023-07-29T17:14:14.105463+0000 mon.a (mon.0) 190 : audit 0 from='client.34326 172.21.15.78:0/4281746270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:15.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[60360]: audit 2023-07-29T17:14:13.290820+0000 mgr.y (mgr.25169) 100 : audit 0 from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:15.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[60360]: audit 2023-07-29T17:14:13.651802+0000 mgr.y (mgr.25169) 101 : audit 0 from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:15.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[60360]: cluster 2023-07-29T17:14:14.030136+0000 mgr.y (mgr.25169) 102 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:15.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[60360]: audit 2023-07-29T17:14:14.105463+0000 mon.a (mon.0) 190 : audit 0 from='client.34326 172.21.15.78:0/4281746270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:15.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[58614]: audit 2023-07-29T17:14:13.290820+0000 mgr.y (mgr.25169) 100 : audit 0 from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:15.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[58614]: audit 2023-07-29T17:14:13.651802+0000 mgr.y (mgr.25169) 101 : audit 0 from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:15.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[58614]: cluster 2023-07-29T17:14:14.030136+0000 mgr.y (mgr.25169) 102 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:15.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:14 smithi078 bash[58614]: audit 2023-07-29T17:14:14.105463+0000 mon.a (mon.0) 190 : audit 0 from='client.34326 172.21.15.78:0/4281746270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:16.097 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.473691+0000 mgr.y (mgr.25169) 103 : audit 0 from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:16.097 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.743630+0000 mon.a (mon.0) 191 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.752799+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: cephadm 2023-07-29T17:14:14.764966+0000 mgr.y (mgr.25169) 104 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.765429+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.766695+0000 mon.a (mon.0) 194 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.767709+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: cephadm 2023-07-29T17:14:14.768924+0000 mgr.y (mgr.25169) 105 : cephadm 1 Reconfiguring daemon mgr.x on smithi129 2023-07-29T17:14:16.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:15 smithi129 bash[53756]: audit 2023-07-29T17:14:14.914931+0000 mon.a (mon.0) 196 : audit 0 from='client.34338 172.21.15.78:0/2906600172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:14:16.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.473691+0000 mgr.y (mgr.25169) 103 : audit 0 from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:16.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.743630+0000 mon.a (mon.0) 191 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:16.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.752799+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:16.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: cephadm 2023-07-29T17:14:14.764966+0000 mgr.y (mgr.25169) 104 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-07-29T17:14:16.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.765429+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:14:16.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.766695+0000 mon.a (mon.0) 194 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:14:16.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.767709+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:16.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: cephadm 2023-07-29T17:14:14.768924+0000 mgr.y (mgr.25169) 105 : cephadm 1 Reconfiguring daemon mgr.x on smithi129 2023-07-29T17:14:16.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[58614]: audit 2023-07-29T17:14:14.914931+0000 mon.a (mon.0) 196 : audit 0 from='client.34338 172.21.15.78:0/2906600172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.473691+0000 mgr.y (mgr.25169) 103 : audit 0 from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.743630+0000 mon.a (mon.0) 191 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.752799+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: cephadm 2023-07-29T17:14:14.764966+0000 mgr.y (mgr.25169) 104 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.765429+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.766695+0000 mon.a (mon.0) 194 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T17:14:16.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.767709+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:16.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: cephadm 2023-07-29T17:14:14.768924+0000 mgr.y (mgr.25169) 105 : cephadm 1 Reconfiguring daemon mgr.x on smithi129 2023-07-29T17:14:16.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:15 smithi078 bash[60360]: audit 2023-07-29T17:14:14.914931+0000 mon.a (mon.0) 196 : audit 0 from='client.34338 172.21.15.78:0/2906600172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:14:17.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:16 smithi129 bash[53756]: cluster 2023-07-29T17:14:16.031832+0000 mgr.y (mgr.25169) 106 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:17.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:16 smithi078 bash[58614]: cluster 2023-07-29T17:14:16.031832+0000 mgr.y (mgr.25169) 106 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:17.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:16 smithi078 bash[60360]: cluster 2023-07-29T17:14:16.031832+0000 mgr.y (mgr.25169) 106 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:18.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:18 smithi129 bash[53756]: audit 2023-07-29T17:14:17.052536+0000 mon.a (mon.0) 197 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:18.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:18 smithi129 bash[53756]: audit 2023-07-29T17:14:17.062654+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:18.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:18 smithi129 bash[53756]: cephadm 2023-07-29T17:14:17.089448+0000 mgr.y (mgr.25169) 107 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-07-29T17:14:18.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:18 smithi129 bash[53756]: audit 2023-07-29T17:14:17.090143+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:14:18.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:18 smithi129 bash[53756]: audit 2023-07-29T17:14:17.091963+0000 mon.a (mon.0) 200 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:18.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:18 smithi129 bash[53756]: cephadm 2023-07-29T17:14:17.097284+0000 mgr.y (mgr.25169) 108 : cephadm 1 Reconfiguring daemon osd.4 on smithi129 2023-07-29T17:14:18.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[58614]: audit 2023-07-29T17:14:17.052536+0000 mon.a (mon.0) 197 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:18.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[58614]: audit 2023-07-29T17:14:17.062654+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:18.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[58614]: cephadm 2023-07-29T17:14:17.089448+0000 mgr.y (mgr.25169) 107 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-07-29T17:14:18.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[58614]: audit 2023-07-29T17:14:17.090143+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:14:18.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[58614]: audit 2023-07-29T17:14:17.091963+0000 mon.a (mon.0) 200 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:18.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[58614]: cephadm 2023-07-29T17:14:17.097284+0000 mgr.y (mgr.25169) 108 : cephadm 1 Reconfiguring daemon osd.4 on smithi129 2023-07-29T17:14:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[60360]: audit 2023-07-29T17:14:17.052536+0000 mon.a (mon.0) 197 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[60360]: audit 2023-07-29T17:14:17.062654+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[60360]: cephadm 2023-07-29T17:14:17.089448+0000 mgr.y (mgr.25169) 107 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-07-29T17:14:18.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[60360]: audit 2023-07-29T17:14:17.090143+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:14:18.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[60360]: audit 2023-07-29T17:14:17.091963+0000 mon.a (mon.0) 200 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:18.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:18 smithi078 bash[60360]: cephadm 2023-07-29T17:14:17.097284+0000 mgr.y (mgr.25169) 108 : cephadm 1 Reconfiguring daemon osd.4 on smithi129 2023-07-29T17:14:19.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:19 smithi129 bash[53756]: cluster 2023-07-29T17:14:18.032562+0000 mgr.y (mgr.25169) 109 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:19.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:19 smithi078 bash[58614]: cluster 2023-07-29T17:14:18.032562+0000 mgr.y (mgr.25169) 109 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:19.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:19 smithi078 bash[60360]: cluster 2023-07-29T17:14:18.032562+0000 mgr.y (mgr.25169) 109 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:20.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: audit 2023-07-29T17:14:19.378655+0000 mon.a (mon.0) 201 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:20.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: audit 2023-07-29T17:14:19.387042+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: cephadm 2023-07-29T17:14:19.416523+0000 mgr.y (mgr.25169) 110 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-07-29T17:14:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: audit 2023-07-29T17:14:19.417115+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:14:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: audit 2023-07-29T17:14:19.418962+0000 mon.a (mon.0) 204 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: cephadm 2023-07-29T17:14:19.424142+0000 mgr.y (mgr.25169) 111 : cephadm 1 Reconfiguring daemon osd.5 on smithi129 2023-07-29T17:14:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:20 smithi129 bash[53756]: cluster 2023-07-29T17:14:20.034248+0000 mgr.y (mgr.25169) 112 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:20.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: audit 2023-07-29T17:14:19.378655+0000 mon.a (mon.0) 201 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:20.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: audit 2023-07-29T17:14:19.387042+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:20.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: cephadm 2023-07-29T17:14:19.416523+0000 mgr.y (mgr.25169) 110 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-07-29T17:14:20.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: audit 2023-07-29T17:14:19.417115+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:14:20.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: audit 2023-07-29T17:14:19.418962+0000 mon.a (mon.0) 204 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:20.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: cephadm 2023-07-29T17:14:19.424142+0000 mgr.y (mgr.25169) 111 : cephadm 1 Reconfiguring daemon osd.5 on smithi129 2023-07-29T17:14:20.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[58614]: cluster 2023-07-29T17:14:20.034248+0000 mgr.y (mgr.25169) 112 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:20.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: audit 2023-07-29T17:14:19.378655+0000 mon.a (mon.0) 201 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:20.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: audit 2023-07-29T17:14:19.387042+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:20.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: cephadm 2023-07-29T17:14:19.416523+0000 mgr.y (mgr.25169) 110 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-07-29T17:14:20.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: audit 2023-07-29T17:14:19.417115+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:14:20.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: audit 2023-07-29T17:14:19.418962+0000 mon.a (mon.0) 204 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:20.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: cephadm 2023-07-29T17:14:19.424142+0000 mgr.y (mgr.25169) 111 : cephadm 1 Reconfiguring daemon osd.5 on smithi129 2023-07-29T17:14:20.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:20 smithi078 bash[60360]: cluster 2023-07-29T17:14:20.034248+0000 mgr.y (mgr.25169) 112 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:23.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: audit 2023-07-29T17:14:21.809057+0000 mon.a (mon.0) 205 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:23.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: audit 2023-07-29T17:14:21.820179+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:23.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: cephadm 2023-07-29T17:14:21.843684+0000 mgr.y (mgr.25169) 113 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-07-29T17:14:23.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: audit 2023-07-29T17:14:21.844324+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:14:23.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: audit 2023-07-29T17:14:21.846019+0000 mon.a (mon.0) 208 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:23.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: cephadm 2023-07-29T17:14:21.851286+0000 mgr.y (mgr.25169) 114 : cephadm 1 Reconfiguring daemon osd.6 on smithi129 2023-07-29T17:14:23.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:22 smithi129 bash[53756]: cluster 2023-07-29T17:14:22.034946+0000 mgr.y (mgr.25169) 115 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:23.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: audit 2023-07-29T17:14:21.809057+0000 mon.a (mon.0) 205 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:23.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: audit 2023-07-29T17:14:21.820179+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:23.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: cephadm 2023-07-29T17:14:21.843684+0000 mgr.y (mgr.25169) 113 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-07-29T17:14:23.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: audit 2023-07-29T17:14:21.844324+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:14:23.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: audit 2023-07-29T17:14:21.846019+0000 mon.a (mon.0) 208 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:23.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: cephadm 2023-07-29T17:14:21.851286+0000 mgr.y (mgr.25169) 114 : cephadm 1 Reconfiguring daemon osd.6 on smithi129 2023-07-29T17:14:23.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[60360]: cluster 2023-07-29T17:14:22.034946+0000 mgr.y (mgr.25169) 115 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:23.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: audit 2023-07-29T17:14:21.809057+0000 mon.a (mon.0) 205 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: audit 2023-07-29T17:14:21.820179+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: cephadm 2023-07-29T17:14:21.843684+0000 mgr.y (mgr.25169) 113 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-07-29T17:14:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: audit 2023-07-29T17:14:21.844324+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:14:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: audit 2023-07-29T17:14:21.846019+0000 mon.a (mon.0) 208 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: cephadm 2023-07-29T17:14:21.851286+0000 mgr.y (mgr.25169) 114 : cephadm 1 Reconfiguring daemon osd.6 on smithi129 2023-07-29T17:14:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:22 smithi078 bash[58614]: cluster 2023-07-29T17:14:22.034946+0000 mgr.y (mgr.25169) 115 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:25.160 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:25 smithi129 bash[53756]: cluster 2023-07-29T17:14:24.035726+0000 mgr.y (mgr.25169) 116 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:25.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:25 smithi129 bash[53756]: audit 2023-07-29T17:14:24.140920+0000 mon.a (mon.0) 209 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:25 smithi129 bash[53756]: audit 2023-07-29T17:14:24.150421+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:25 smithi129 bash[53756]: audit 2023-07-29T17:14:24.171099+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:14:25.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:25 smithi129 bash[53756]: audit 2023-07-29T17:14:24.172901+0000 mon.a (mon.0) 212 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:25.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[60360]: cluster 2023-07-29T17:14:24.035726+0000 mgr.y (mgr.25169) 116 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[60360]: audit 2023-07-29T17:14:24.140920+0000 mon.a (mon.0) 209 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[60360]: audit 2023-07-29T17:14:24.150421+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[60360]: audit 2023-07-29T17:14:24.171099+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:14:25.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[60360]: audit 2023-07-29T17:14:24.172901+0000 mon.a (mon.0) 212 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:25.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[58614]: cluster 2023-07-29T17:14:24.035726+0000 mgr.y (mgr.25169) 116 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:25.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[58614]: audit 2023-07-29T17:14:24.140920+0000 mon.a (mon.0) 209 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:25.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[58614]: audit 2023-07-29T17:14:24.150421+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:25.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[58614]: audit 2023-07-29T17:14:24.171099+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:14:25.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:25 smithi078 bash[58614]: audit 2023-07-29T17:14:24.172901+0000 mon.a (mon.0) 212 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:26.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:26 smithi129 bash[53756]: cephadm 2023-07-29T17:14:24.170458+0000 mgr.y (mgr.25169) 117 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-07-29T17:14:26.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:26 smithi129 bash[53756]: cephadm 2023-07-29T17:14:24.178210+0000 mgr.y (mgr.25169) 118 : cephadm 1 Reconfiguring daemon osd.7 on smithi129 2023-07-29T17:14:26.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:26 smithi078 bash[58614]: cephadm 2023-07-29T17:14:24.170458+0000 mgr.y (mgr.25169) 117 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-07-29T17:14:26.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:26 smithi078 bash[58614]: cephadm 2023-07-29T17:14:24.178210+0000 mgr.y (mgr.25169) 118 : cephadm 1 Reconfiguring daemon osd.7 on smithi129 2023-07-29T17:14:26.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:26 smithi078 bash[60360]: cephadm 2023-07-29T17:14:24.170458+0000 mgr.y (mgr.25169) 117 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-07-29T17:14:26.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:26 smithi078 bash[60360]: cephadm 2023-07-29T17:14:24.178210+0000 mgr.y (mgr.25169) 118 : cephadm 1 Reconfiguring daemon osd.7 on smithi129 2023-07-29T17:14:27.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[58614]: cluster 2023-07-29T17:14:26.037318+0000 mgr.y (mgr.25169) 119 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:27.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[58614]: audit 2023-07-29T17:14:26.428522+0000 mon.a (mon.0) 213 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:27.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[58614]: audit 2023-07-29T17:14:26.439251+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:27.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[58614]: audit 2023-07-29T17:14:26.448823+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:27.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[58614]: audit 2023-07-29T17:14:26.452168+0000 mon.a (mon.0) 216 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:27.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[60360]: cluster 2023-07-29T17:14:26.037318+0000 mgr.y (mgr.25169) 119 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:27.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[60360]: audit 2023-07-29T17:14:26.428522+0000 mon.a (mon.0) 213 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:27.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[60360]: audit 2023-07-29T17:14:26.439251+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:27.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[60360]: audit 2023-07-29T17:14:26.448823+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:27.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:27 smithi078 bash[60360]: audit 2023-07-29T17:14:26.452168+0000 mon.a (mon.0) 216 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:27.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:27 smithi129 bash[53756]: cluster 2023-07-29T17:14:26.037318+0000 mgr.y (mgr.25169) 119 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:27.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:27 smithi129 bash[53756]: audit 2023-07-29T17:14:26.428522+0000 mon.a (mon.0) 213 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:27.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:27 smithi129 bash[53756]: audit 2023-07-29T17:14:26.439251+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:27.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:27 smithi129 bash[53756]: audit 2023-07-29T17:14:26.448823+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:27.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:27 smithi129 bash[53756]: audit 2023-07-29T17:14:26.452168+0000 mon.a (mon.0) 216 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:28.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:28 smithi078 bash[60360]: cephadm 2023-07-29T17:14:26.448199+0000 mgr.y (mgr.25169) 120 : cephadm 1 Reconfiguring rgw.foo.smithi129.dbexjf (monmap changed)... 2023-07-29T17:14:28.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:28 smithi078 bash[60360]: cephadm 2023-07-29T17:14:26.453981+0000 mgr.y (mgr.25169) 121 : cephadm 1 Reconfiguring daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:14:28.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:28 smithi078 bash[58614]: cephadm 2023-07-29T17:14:26.448199+0000 mgr.y (mgr.25169) 120 : cephadm 1 Reconfiguring rgw.foo.smithi129.dbexjf (monmap changed)... 2023-07-29T17:14:28.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:28 smithi078 bash[58614]: cephadm 2023-07-29T17:14:26.453981+0000 mgr.y (mgr.25169) 121 : cephadm 1 Reconfiguring daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:14:28.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:28 smithi129 bash[53756]: cephadm 2023-07-29T17:14:26.448199+0000 mgr.y (mgr.25169) 120 : cephadm 1 Reconfiguring rgw.foo.smithi129.dbexjf (monmap changed)... 2023-07-29T17:14:28.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:28 smithi129 bash[53756]: cephadm 2023-07-29T17:14:26.453981+0000 mgr.y (mgr.25169) 121 : cephadm 1 Reconfiguring daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:14:29.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:29 smithi129 bash[53756]: cluster 2023-07-29T17:14:28.038147+0000 mgr.y (mgr.25169) 122 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:29.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:29 smithi129 bash[53756]: audit 2023-07-29T17:14:28.741708+0000 mon.a (mon.0) 217 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:29.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:29 smithi129 bash[53756]: audit 2023-07-29T17:14:28.752204+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:29.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:29 smithi129 bash[53756]: audit 2023-07-29T17:14:28.775795+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:29.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:29 smithi129 bash[53756]: audit 2023-07-29T17:14:28.779252+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:29.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[60360]: cluster 2023-07-29T17:14:28.038147+0000 mgr.y (mgr.25169) 122 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:29.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[60360]: audit 2023-07-29T17:14:28.741708+0000 mon.a (mon.0) 217 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:29.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[60360]: audit 2023-07-29T17:14:28.752204+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:29.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[60360]: audit 2023-07-29T17:14:28.775795+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:29.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[60360]: audit 2023-07-29T17:14:28.779252+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:29.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[58614]: cluster 2023-07-29T17:14:28.038147+0000 mgr.y (mgr.25169) 122 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:29.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[58614]: audit 2023-07-29T17:14:28.741708+0000 mon.a (mon.0) 217 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:29.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[58614]: audit 2023-07-29T17:14:28.752204+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:29.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[58614]: audit 2023-07-29T17:14:28.775795+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:14:29.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:29 smithi078 bash[58614]: audit 2023-07-29T17:14:28.779252+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:30.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:30 smithi078 bash[58614]: cephadm 2023-07-29T17:14:28.775161+0000 mgr.y (mgr.25169) 123 : cephadm 1 Reconfiguring rgw.smpl.smithi129.tqsnpp (monmap changed)... 2023-07-29T17:14:30.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:30 smithi078 bash[58614]: cephadm 2023-07-29T17:14:28.781125+0000 mgr.y (mgr.25169) 124 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:14:30.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:30 smithi078 bash[60360]: cephadm 2023-07-29T17:14:28.775161+0000 mgr.y (mgr.25169) 123 : cephadm 1 Reconfiguring rgw.smpl.smithi129.tqsnpp (monmap changed)... 2023-07-29T17:14:30.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:30 smithi078 bash[60360]: cephadm 2023-07-29T17:14:28.781125+0000 mgr.y (mgr.25169) 124 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:14:30.537 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:30 smithi129 bash[53756]: cephadm 2023-07-29T17:14:28.775161+0000 mgr.y (mgr.25169) 123 : cephadm 1 Reconfiguring rgw.smpl.smithi129.tqsnpp (monmap changed)... 2023-07-29T17:14:30.538 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:30 smithi129 bash[53756]: cephadm 2023-07-29T17:14:28.781125+0000 mgr.y (mgr.25169) 124 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:14:31.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[60360]: cluster 2023-07-29T17:14:30.039984+0000 mgr.y (mgr.25169) 125 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:31.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[60360]: audit 2023-07-29T17:14:31.109627+0000 mon.a (mon.0) 221 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:31.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[60360]: audit 2023-07-29T17:14:31.121995+0000 mon.a (mon.0) 222 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:31.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[60360]: audit 2023-07-29T17:14:31.164494+0000 mon.a (mon.0) 223 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:14:31.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[60360]: audit 2023-07-29T17:14:31.168918+0000 mon.a (mon.0) 224 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:31.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[58614]: cluster 2023-07-29T17:14:30.039984+0000 mgr.y (mgr.25169) 125 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:31.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[58614]: audit 2023-07-29T17:14:31.109627+0000 mon.a (mon.0) 221 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:31.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[58614]: audit 2023-07-29T17:14:31.121995+0000 mon.a (mon.0) 222 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:31.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[58614]: audit 2023-07-29T17:14:31.164494+0000 mon.a (mon.0) 223 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:14:31.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:31 smithi078 bash[58614]: audit 2023-07-29T17:14:31.168918+0000 mon.a (mon.0) 224 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:31.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:31 smithi129 bash[53756]: cluster 2023-07-29T17:14:30.039984+0000 mgr.y (mgr.25169) 125 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:31 smithi129 bash[53756]: audit 2023-07-29T17:14:31.109627+0000 mon.a (mon.0) 221 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:31 smithi129 bash[53756]: audit 2023-07-29T17:14:31.121995+0000 mon.a (mon.0) 222 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:31.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:31 smithi129 bash[53756]: audit 2023-07-29T17:14:31.164494+0000 mon.a (mon.0) 223 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:14:31.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:31 smithi129 bash[53756]: audit 2023-07-29T17:14:31.168918+0000 mon.a (mon.0) 224 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.171474+0000 mon.a (mon.0) 225 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: cephadm 2023-07-29T17:14:31.173097+0000 mgr.y (mgr.25169) 126 : cephadm 1 Upgrade: Setting container_image for all mon 2023-07-29T17:14:32.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.186415+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.187891+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.192324+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.193659+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.200524+0000 mon.a (mon.0) 230 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.201897+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.207926+0000 mon.a (mon.0) 232 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-07-29T17:14:32.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.209819+0000 mon.a (mon.0) 233 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: cephadm 2023-07-29T17:14:31.210917+0000 mgr.y (mgr.25169) 127 : cephadm 1 Upgrade: Setting container_image for all crash 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.218690+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.221084+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: audit 2023-07-29T17:14:31.221345+0000 mgr.y (mgr.25169) 128 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: cephadm 2023-07-29T17:14:31.223207+0000 mgr.y (mgr.25169) 129 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[58614]: cluster 2023-07-29T17:14:32.040724+0000 mgr.y (mgr.25169) 130 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:32.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.171474+0000 mon.a (mon.0) 225 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: cephadm 2023-07-29T17:14:31.173097+0000 mgr.y (mgr.25169) 126 : cephadm 1 Upgrade: Setting container_image for all mon 2023-07-29T17:14:32.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.186415+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:32.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.187891+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-07-29T17:14:32.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.192324+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-07-29T17:14:32.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.193659+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-07-29T17:14:32.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.200524+0000 mon.a (mon.0) 230 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.201897+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.207926+0000 mon.a (mon.0) 232 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.209819+0000 mon.a (mon.0) 233 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: cephadm 2023-07-29T17:14:31.210917+0000 mgr.y (mgr.25169) 127 : cephadm 1 Upgrade: Setting container_image for all crash 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.218690+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.221084+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: audit 2023-07-29T17:14:31.221345+0000 mgr.y (mgr.25169) 128 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-29T17:14:32.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: cephadm 2023-07-29T17:14:31.223207+0000 mgr.y (mgr.25169) 129 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-07-29T17:14:32.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:32 smithi078 bash[60360]: cluster 2023-07-29T17:14:32.040724+0000 mgr.y (mgr.25169) 130 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:32.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.171474+0000 mon.a (mon.0) 225 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: cephadm 2023-07-29T17:14:31.173097+0000 mgr.y (mgr.25169) 126 : cephadm 1 Upgrade: Setting container_image for all mon 2023-07-29T17:14:32.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.186415+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:32.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.187891+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-07-29T17:14:32.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.192324+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-07-29T17:14:32.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.193659+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-07-29T17:14:32.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.200524+0000 mon.a (mon.0) 230 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-07-29T17:14:32.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.201897+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-07-29T17:14:32.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.207926+0000 mon.a (mon.0) 232 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-07-29T17:14:32.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.209819+0000 mon.a (mon.0) 233 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:32.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: cephadm 2023-07-29T17:14:31.210917+0000 mgr.y (mgr.25169) 127 : cephadm 1 Upgrade: Setting container_image for all crash 2023-07-29T17:14:32.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.218690+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:32.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.221084+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-29T17:14:32.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: audit 2023-07-29T17:14:31.221345+0000 mgr.y (mgr.25169) 128 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-29T17:14:32.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: cephadm 2023-07-29T17:14:31.223207+0000 mgr.y (mgr.25169) 129 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-07-29T17:14:32.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:32 smithi129 bash[53756]: cluster 2023-07-29T17:14:32.040724+0000 mgr.y (mgr.25169) 130 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:34.848 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[58614]: cephadm 2023-07-29T17:14:33.543872+0000 mgr.y (mgr.25169) 131 : cephadm 1 Upgrade: Updating osd.0 2023-07-29T17:14:34.848 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[58614]: audit 2023-07-29T17:14:33.553823+0000 mon.a (mon.0) 236 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:34.848 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[58614]: audit 2023-07-29T17:14:33.555438+0000 mon.a (mon.0) 237 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T17:14:34.848 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[58614]: audit 2023-07-29T17:14:33.556684+0000 mon.a (mon.0) 238 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:34.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[58614]: cephadm 2023-07-29T17:14:33.561553+0000 mgr.y (mgr.25169) 132 : cephadm 1 Deploying daemon osd.0 on smithi078 2023-07-29T17:14:34.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[58614]: cluster 2023-07-29T17:14:34.041553+0000 mgr.y (mgr.25169) 133 : cluster 0 pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:34.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[60360]: cephadm 2023-07-29T17:14:33.543872+0000 mgr.y (mgr.25169) 131 : cephadm 1 Upgrade: Updating osd.0 2023-07-29T17:14:34.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[60360]: audit 2023-07-29T17:14:33.553823+0000 mon.a (mon.0) 236 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:34.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[60360]: audit 2023-07-29T17:14:33.555438+0000 mon.a (mon.0) 237 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T17:14:34.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[60360]: audit 2023-07-29T17:14:33.556684+0000 mon.a (mon.0) 238 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:34.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[60360]: cephadm 2023-07-29T17:14:33.561553+0000 mgr.y (mgr.25169) 132 : cephadm 1 Deploying daemon osd.0 on smithi078 2023-07-29T17:14:34.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:34 smithi078 bash[60360]: cluster 2023-07-29T17:14:34.041553+0000 mgr.y (mgr.25169) 133 : cluster 0 pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:34.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:34 smithi129 bash[53756]: cephadm 2023-07-29T17:14:33.543872+0000 mgr.y (mgr.25169) 131 : cephadm 1 Upgrade: Updating osd.0 2023-07-29T17:14:34.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:34 smithi129 bash[53756]: audit 2023-07-29T17:14:33.553823+0000 mon.a (mon.0) 236 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:34.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:34 smithi129 bash[53756]: audit 2023-07-29T17:14:33.555438+0000 mon.a (mon.0) 237 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T17:14:34.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:34 smithi129 bash[53756]: audit 2023-07-29T17:14:33.556684+0000 mon.a (mon.0) 238 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:14:34.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:34 smithi129 bash[53756]: cephadm 2023-07-29T17:14:33.561553+0000 mgr.y (mgr.25169) 132 : cephadm 1 Deploying daemon osd.0 on smithi078 2023-07-29T17:14:34.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:34 smithi129 bash[53756]: cluster 2023-07-29T17:14:34.041553+0000 mgr.y (mgr.25169) 133 : cluster 0 pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:37.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:37 smithi129 bash[53756]: cluster 2023-07-29T17:14:36.043249+0000 mgr.y (mgr.25169) 134 : cluster 0 pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:37.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:37 smithi078 bash[58614]: cluster 2023-07-29T17:14:36.043249+0000 mgr.y (mgr.25169) 134 : cluster 0 pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:37.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:37 smithi078 bash[60360]: cluster 2023-07-29T17:14:36.043249+0000 mgr.y (mgr.25169) 134 : cluster 0 pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:38.102 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:37 smithi078 systemd[1]: Stopping Ceph osd.0 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:14:38.441 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:38 smithi078 bash[30909]: debug 2023-07-29T17:14:38.076+0000 7f5a608aa700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:14:38.442 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:38 smithi078 bash[30909]: debug 2023-07-29T17:14:38.076+0000 7f5a608aa700 -1 osd.0 78 *** Got signal Terminated *** 2023-07-29T17:14:38.442 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:38 smithi078 bash[30909]: debug 2023-07-29T17:14:38.076+0000 7f5a608aa700 -1 osd.0 78 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:14:39.191 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:38 smithi078 bash[62879]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.0 2023-07-29T17:14:39.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.044004+0000 mgr.y (mgr.25169) 135 : cluster 0 pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:39.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.275072+0000 mon.a (mon.0) 239 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.275138+0000 mon.a (mon.0) 240 : cluster 1 osd.0 failed (root=default,host=smithi078) (connection refused reported by osd.7) 2023-07-29T17:14:39.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.278571+0000 mon.a (mon.0) 241 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.280657+0000 mon.a (mon.0) 242 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.282919+0000 mon.a (mon.0) 243 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.350025+0000 mon.a (mon.0) 244 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.350528+0000 mon.a (mon.0) 245 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.350991+0000 mon.a (mon.0) 246 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.351426+0000 mon.a (mon.0) 247 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.352516+0000 mon.a (mon.0) 248 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.352960+0000 mon.a (mon.0) 249 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.353495+0000 mon.a (mon.0) 250 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.353889+0000 mon.a (mon.0) 251 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.354186+0000 mon.a (mon.0) 252 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.354405+0000 mon.a (mon.0) 253 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.354717+0000 mon.a (mon.0) 254 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.354949+0000 mon.a (mon.0) 255 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.355319+0000 mon.a (mon.0) 256 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.355563+0000 mon.a (mon.0) 257 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.477412+0000 mon.a (mon.0) 258 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.480093+0000 mon.a (mon.0) 259 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.481968+0000 mon.a (mon.0) 260 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.485998+0000 mon.a (mon.0) 261 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.756314+0000 mon.a (mon.0) 262 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.756869+0000 mon.a (mon.0) 263 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.044004+0000 mgr.y (mgr.25169) 135 : cluster 0 pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:39.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.275072+0000 mon.a (mon.0) 239 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.275138+0000 mon.a (mon.0) 240 : cluster 1 osd.0 failed (root=default,host=smithi078) (connection refused reported by osd.7) 2023-07-29T17:14:39.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.278571+0000 mon.a (mon.0) 241 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.280657+0000 mon.a (mon.0) 242 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.282919+0000 mon.a (mon.0) 243 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.350025+0000 mon.a (mon.0) 244 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.350528+0000 mon.a (mon.0) 245 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.350991+0000 mon.a (mon.0) 246 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.351426+0000 mon.a (mon.0) 247 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.352516+0000 mon.a (mon.0) 248 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.352960+0000 mon.a (mon.0) 249 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.353495+0000 mon.a (mon.0) 250 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.353889+0000 mon.a (mon.0) 251 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.354186+0000 mon.a (mon.0) 252 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.354405+0000 mon.a (mon.0) 253 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.354717+0000 mon.a (mon.0) 254 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.354949+0000 mon.a (mon.0) 255 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.355319+0000 mon.a (mon.0) 256 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.355563+0000 mon.a (mon.0) 257 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.477412+0000 mon.a (mon.0) 258 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.480093+0000 mon.a (mon.0) 259 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.481968+0000 mon.a (mon.0) 260 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.485998+0000 mon.a (mon.0) 261 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.756314+0000 mon.a (mon.0) 262 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.756869+0000 mon.a (mon.0) 263 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.757456+0000 mon.a (mon.0) 264 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.757836+0000 mon.a (mon.0) 265 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.757960+0000 mon.a (mon.0) 266 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.758242+0000 mon.a (mon.0) 267 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.758437+0000 mon.a (mon.0) 268 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.758742+0000 mon.a (mon.0) 269 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.758889+0000 mon.a (mon.0) 270 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.759180+0000 mon.a (mon.0) 271 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.759370+0000 mon.a (mon.0) 272 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.759718+0000 mon.a (mon.0) 273 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.760009+0000 mon.a (mon.0) 274 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.760281+0000 mon.a (mon.0) 275 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.881575+0000 mon.a (mon.0) 276 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.884125+0000 mon.a (mon.0) 277 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.886090+0000 mon.a (mon.0) 278 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[60360]: cluster 2023-07-29T17:14:38.890072+0000 mon.a (mon.0) 279 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.757456+0000 mon.a (mon.0) 264 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.757836+0000 mon.a (mon.0) 265 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.757960+0000 mon.a (mon.0) 266 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.758242+0000 mon.a (mon.0) 267 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.758437+0000 mon.a (mon.0) 268 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.758742+0000 mon.a (mon.0) 269 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.758889+0000 mon.a (mon.0) 270 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.759180+0000 mon.a (mon.0) 271 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.759370+0000 mon.a (mon.0) 272 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.759718+0000 mon.a (mon.0) 273 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.760009+0000 mon.a (mon.0) 274 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.760281+0000 mon.a (mon.0) 275 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.881575+0000 mon.a (mon.0) 276 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.884125+0000 mon.a (mon.0) 277 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.886090+0000 mon.a (mon.0) 278 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:39 smithi078 bash[58614]: cluster 2023-07-29T17:14:38.890072+0000 mon.a (mon.0) 279 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.044004+0000 mgr.y (mgr.25169) 135 : cluster 0 pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:39.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.275072+0000 mon.a (mon.0) 239 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.275138+0000 mon.a (mon.0) 240 : cluster 1 osd.0 failed (root=default,host=smithi078) (connection refused reported by osd.7) 2023-07-29T17:14:39.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.278571+0000 mon.a (mon.0) 241 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.280657+0000 mon.a (mon.0) 242 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.282919+0000 mon.a (mon.0) 243 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.350025+0000 mon.a (mon.0) 244 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.350528+0000 mon.a (mon.0) 245 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.350991+0000 mon.a (mon.0) 246 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.351426+0000 mon.a (mon.0) 247 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.352516+0000 mon.a (mon.0) 248 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.352960+0000 mon.a (mon.0) 249 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.353495+0000 mon.a (mon.0) 250 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.353889+0000 mon.a (mon.0) 251 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.354186+0000 mon.a (mon.0) 252 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.354405+0000 mon.a (mon.0) 253 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.354717+0000 mon.a (mon.0) 254 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.354949+0000 mon.a (mon.0) 255 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.355319+0000 mon.a (mon.0) 256 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.355563+0000 mon.a (mon.0) 257 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.477412+0000 mon.a (mon.0) 258 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.480093+0000 mon.a (mon.0) 259 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.481968+0000 mon.a (mon.0) 260 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.485998+0000 mon.a (mon.0) 261 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.756314+0000 mon.a (mon.0) 262 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.756869+0000 mon.a (mon.0) 263 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.757456+0000 mon.a (mon.0) 264 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.757836+0000 mon.a (mon.0) 265 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.757960+0000 mon.a (mon.0) 266 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.758242+0000 mon.a (mon.0) 267 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-29T17:14:39.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.758437+0000 mon.a (mon.0) 268 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.758742+0000 mon.a (mon.0) 269 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.758889+0000 mon.a (mon.0) 270 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.759180+0000 mon.a (mon.0) 271 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.759370+0000 mon.a (mon.0) 272 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.759718+0000 mon.a (mon.0) 273 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.760009+0000 mon.a (mon.0) 274 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.760281+0000 mon.a (mon.0) 275 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-29T17:14:39.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.881575+0000 mon.a (mon.0) 276 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-29T17:14:39.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.884125+0000 mon.a (mon.0) 277 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-29T17:14:39.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.886090+0000 mon.a (mon.0) 278 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-29T17:14:39.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:39 smithi129 bash[53756]: cluster 2023-07-29T17:14:38.890072+0000 mon.a (mon.0) 279 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-29T17:14:40.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:40 smithi129 bash[53756]: cluster 2023-07-29T17:14:39.104106+0000 mon.a (mon.0) 280 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:14:40.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:40 smithi129 bash[53756]: cluster 2023-07-29T17:14:39.108518+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:40 smithi129 bash[53756]: cluster 2023-07-29T17:14:39.111009+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:40 smithi129 bash[53756]: cluster 2023-07-29T17:14:39.111373+0000 mon.a (mon.0) 281 : cluster 0 osdmap e79: 8 total, 7 up, 8 in 2023-07-29T17:14:40.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:40 smithi129 bash[53756]: cluster 2023-07-29T17:14:39.112958+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:40 smithi129 bash[53756]: cluster 2023-07-29T17:14:39.115775+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[60360]: cluster 2023-07-29T17:14:39.104106+0000 mon.a (mon.0) 280 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:14:40.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[60360]: cluster 2023-07-29T17:14:39.108518+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[60360]: cluster 2023-07-29T17:14:39.111009+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[60360]: cluster 2023-07-29T17:14:39.111373+0000 mon.a (mon.0) 281 : cluster 0 osdmap e79: 8 total, 7 up, 8 in 2023-07-29T17:14:40.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[60360]: cluster 2023-07-29T17:14:39.112958+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[60360]: cluster 2023-07-29T17:14:39.115775+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[58614]: cluster 2023-07-29T17:14:39.104106+0000 mon.a (mon.0) 280 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:14:40.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[58614]: cluster 2023-07-29T17:14:39.108518+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[58614]: cluster 2023-07-29T17:14:39.111009+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[58614]: cluster 2023-07-29T17:14:39.111373+0000 mon.a (mon.0) 281 : cluster 0 osdmap e79: 8 total, 7 up, 8 in 2023-07-29T17:14:40.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[58614]: cluster 2023-07-29T17:14:39.112958+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:40.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:40 smithi078 bash[58614]: cluster 2023-07-29T17:14:39.115775+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-29T17:14:41.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.046391+0000 mgr.y (mgr.25169) 136 : cluster 0 pgmap v70: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:41.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.114639+0000 mon.a (mon.0) 282 : cluster 3 Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2023-07-29T17:14:41.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.130832+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.133770+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.134213+0000 mon.a (mon.0) 283 : cluster 0 osdmap e80: 8 total, 7 up, 8 in 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.134769+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.134830+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.137124+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.137184+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.137288+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.139707+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:41 smithi129 bash[53756]: cluster 2023-07-29T17:14:40.139937+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.441 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:41 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.0.service: Succeeded. 2023-07-29T17:14:41.441 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:41 smithi078 systemd[1]: Stopped Ceph osd.0 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:14:41.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.046391+0000 mgr.y (mgr.25169) 136 : cluster 0 pgmap v70: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:41.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.114639+0000 mon.a (mon.0) 282 : cluster 3 Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2023-07-29T17:14:41.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.130832+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.133770+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.134213+0000 mon.a (mon.0) 283 : cluster 0 osdmap e80: 8 total, 7 up, 8 in 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.134769+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.134830+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.137124+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.137184+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.137288+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.139707+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[58614]: cluster 2023-07-29T17:14:40.139937+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.046391+0000 mgr.y (mgr.25169) 136 : cluster 0 pgmap v70: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:41.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.114639+0000 mon.a (mon.0) 282 : cluster 3 Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2023-07-29T17:14:41.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.130832+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.133770+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.134213+0000 mon.a (mon.0) 283 : cluster 0 osdmap e80: 8 total, 7 up, 8 in 2023-07-29T17:14:41.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.134769+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.134830+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.137124+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.137184+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.137288+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.139707+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:41 smithi078 bash[60360]: cluster 2023-07-29T17:14:40.139937+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e80 with expected crc 2023-07-29T17:14:41.943 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:41 smithi078 systemd[1]: Started Ceph osd.0 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:14:42.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:42 smithi129 bash[53756]: audit 2023-07-29T17:14:41.553377+0000 mon.a (mon.0) 284 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:42.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:42 smithi129 bash[53756]: audit 2023-07-29T17:14:41.560289+0000 mon.a (mon.0) 285 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:42.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:42 smithi129 bash[53756]: cluster 2023-07-29T17:14:42.047963+0000 mgr.y (mgr.25169) 137 : cluster 0 pgmap v72: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:42.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:42 smithi078 bash[58614]: audit 2023-07-29T17:14:41.553377+0000 mon.a (mon.0) 284 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:42.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:42 smithi078 bash[58614]: audit 2023-07-29T17:14:41.560289+0000 mon.a (mon.0) 285 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:42.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:42 smithi078 bash[58614]: cluster 2023-07-29T17:14:42.047963+0000 mgr.y (mgr.25169) 137 : cluster 0 pgmap v72: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:42.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:42 smithi078 bash[60360]: audit 2023-07-29T17:14:41.553377+0000 mon.a (mon.0) 284 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:42.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:42 smithi078 bash[60360]: audit 2023-07-29T17:14:41.560289+0000 mon.a (mon.0) 285 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:42.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:42 smithi078 bash[60360]: cluster 2023-07-29T17:14:42.047963+0000 mgr.y (mgr.25169) 137 : cluster 0 pgmap v72: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:43.691 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-29T17:14:43.692 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-07-29T17:14:43.692 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-07-29T17:14:43.692 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-29T17:14:43.692 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2023-07-29T17:14:43.692 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-29T17:14:43.692 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:43 smithi078 bash[63160]: --> ceph-volume raw activate successful for osd ID: 0 2023-07-29T17:14:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:44 smithi078 bash[58614]: cluster 2023-07-29T17:14:44.049390+0000 mgr.y (mgr.25169) 138 : cluster 0 pgmap v73: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:44 smithi078 bash[60360]: cluster 2023-07-29T17:14:44.049390+0000 mgr.y (mgr.25169) 138 : cluster 0 pgmap v73: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:45.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:44 smithi129 bash[53756]: cluster 2023-07-29T17:14:44.049390+0000 mgr.y (mgr.25169) 138 : cluster 0 pgmap v73: 161 pgs: 58 peering, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:45.346 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:14:45.441 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:45 smithi078 bash[63468]: debug 2023-07-29T17:14:45.111+0000 7ff504d78700 1 -- 172.21.15.78:0/3494439440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f28ea904e0 con 0x55f28eaeac00 2023-07-29T17:14:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:14:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4m) 91s ago 10m 10.2M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:14:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (3m) 65s ago 10m 26.1M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:14:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 65s ago 8m - - 2023-07-29T17:14:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (2m) 65s ago 15m 428M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (3m) 91s ago 16m 464M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2m) 91s ago 17m 46.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (73s) 65s ago 15m 37.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (104s) 91s ago 15m 35.5M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4m) 91s ago 11m 9148k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (4m) 65s ago 11m 3644k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:14:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 starting - - - 4096M 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (14m) 91s ago 14m 53.4M 4096M 16.2.0 24ecd6d5f14c 322352875a39 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (13m) 91s ago 14m 48.3M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (13m) 91s ago 13m 53.9M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (13m) 65s ago 13m 55.5M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (12m) 65s ago 12m 53.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (12m) 65s ago 12m 53.3M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:14:46.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (11m) 65s ago 11m 56.5M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:14:46.082 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 65s ago 11m - - 2023-07-29T17:14:46.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (8m) 91s ago 8m 54.3M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:14:46.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (8m) 65s ago 8m 54.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:14:46.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (8m) 91s ago 8m 54.7M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:14:46.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (8m) 65s ago 8m 54.5M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:14:46.175 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:45 smithi078 bash[63468]: debug 2023-07-29T17:14:45.667+0000 7ff50defd640 -1 Falling back to public interface 2023-07-29T17:14:46.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:46 smithi078 bash[58614]: cluster 2023-07-29T17:14:46.102719+0000 mon.a (mon.0) 286 : cluster 3 Health check failed: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded (PG_DEGRADED) 2023-07-29T17:14:46.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:46 smithi078 bash[58614]: cluster 2023-07-29T17:14:46.102785+0000 mon.a (mon.0) 287 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2023-07-29T17:14:46.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:46 smithi078 bash[60360]: cluster 2023-07-29T17:14:46.102719+0000 mon.a (mon.0) 286 : cluster 3 Health check failed: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded (PG_DEGRADED) 2023-07-29T17:14:46.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:46 smithi078 bash[60360]: cluster 2023-07-29T17:14:46.102785+0000 mon.a (mon.0) 287 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2023-07-29T17:14:46.520 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:14:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:14:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:14:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:14:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:14:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:14:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 11, 2023-07-29T17:14:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 5 2023-07-29T17:14:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:14:46.523 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:14:46.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:46 smithi129 bash[53756]: cluster 2023-07-29T17:14:46.102719+0000 mon.a (mon.0) 286 : cluster 3 Health check failed: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded (PG_DEGRADED) 2023-07-29T17:14:46.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:46 smithi129 bash[53756]: cluster 2023-07-29T17:14:46.102785+0000 mon.a (mon.0) 287 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2023-07-29T17:14:46.900 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:14:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:14:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:14:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "5/23 daemons upgraded", 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:14:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:14:46.902 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:14:47.335 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded 2023-07-29T17:14:47.336 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:14:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:14:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:14:47.337 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] OSD_DOWN: 1 osds down 2023-07-29T17:14:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: osd.0 (root=default,host=smithi078) is down 2023-07-29T17:14:47.337 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded 2023-07-29T17:14:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2023-07-29T17:14:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2023-07-29T17:14:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.5 is active+undersized+degraded, acting [7,4] 2023-07-29T17:14:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2023-07-29T17:14:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2023-07-29T17:14:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2023-07-29T17:14:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2023-07-29T17:14:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2023-07-29T17:14:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2023-07-29T17:14:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2023-07-29T17:14:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2023-07-29T17:14:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2023-07-29T17:14:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2023-07-29T17:14:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2023-07-29T17:14:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2023-07-29T17:14:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2023-07-29T17:14:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2023-07-29T17:14:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2023-07-29T17:14:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2023-07-29T17:14:47.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[58614]: audit 2023-07-29T17:14:45.306844+0000 mgr.y (mgr.25169) 139 : audit 0 from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[58614]: audit 2023-07-29T17:14:45.684710+0000 mgr.y (mgr.25169) 140 : audit 0 from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[58614]: audit 2023-07-29T17:14:46.037170+0000 mgr.y (mgr.25169) 141 : audit 0 from='client.44189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[58614]: cluster 2023-07-29T17:14:46.050524+0000 mgr.y (mgr.25169) 142 : cluster 0 pgmap v74: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:47.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[58614]: audit 2023-07-29T17:14:46.499911+0000 mon.c (mon.2) 4 : audit 0 from='client.44195 172.21.15.78:0/1110680236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:47.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[60360]: audit 2023-07-29T17:14:45.306844+0000 mgr.y (mgr.25169) 139 : audit 0 from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[60360]: audit 2023-07-29T17:14:45.684710+0000 mgr.y (mgr.25169) 140 : audit 0 from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[60360]: audit 2023-07-29T17:14:46.037170+0000 mgr.y (mgr.25169) 141 : audit 0 from='client.44189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[60360]: cluster 2023-07-29T17:14:46.050524+0000 mgr.y (mgr.25169) 142 : cluster 0 pgmap v74: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:47.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:47 smithi078 bash[60360]: audit 2023-07-29T17:14:46.499911+0000 mon.c (mon.2) 4 : audit 0 from='client.44195 172.21.15.78:0/1110680236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:47.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:47 smithi129 bash[53756]: audit 2023-07-29T17:14:45.306844+0000 mgr.y (mgr.25169) 139 : audit 0 from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:47 smithi129 bash[53756]: audit 2023-07-29T17:14:45.684710+0000 mgr.y (mgr.25169) 140 : audit 0 from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:47 smithi129 bash[53756]: audit 2023-07-29T17:14:46.037170+0000 mgr.y (mgr.25169) 141 : audit 0 from='client.44189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:47.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:47 smithi129 bash[53756]: cluster 2023-07-29T17:14:46.050524+0000 mgr.y (mgr.25169) 142 : cluster 0 pgmap v74: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:47.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:47 smithi129 bash[53756]: audit 2023-07-29T17:14:46.499911+0000 mon.c (mon.2) 4 : audit 0 from='client.44195 172.21.15.78:0/1110680236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:14:48.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:48 smithi078 bash[58614]: audit 2023-07-29T17:14:46.879373+0000 mgr.y (mgr.25169) 143 : audit 0 from='client.44201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:48.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:48 smithi078 bash[58614]: audit 2023-07-29T17:14:47.314934+0000 mon.c (mon.2) 5 : audit 0 from='client.44207 172.21.15.78:0/1378947879' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:14:48.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:48 smithi078 bash[60360]: audit 2023-07-29T17:14:46.879373+0000 mgr.y (mgr.25169) 143 : audit 0 from='client.44201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:48.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:48 smithi078 bash[60360]: audit 2023-07-29T17:14:47.314934+0000 mon.c (mon.2) 5 : audit 0 from='client.44207 172.21.15.78:0/1378947879' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:14:48.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:48 smithi129 bash[53756]: audit 2023-07-29T17:14:46.879373+0000 mgr.y (mgr.25169) 143 : audit 0 from='client.44201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:14:48.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:48 smithi129 bash[53756]: audit 2023-07-29T17:14:47.314934+0000 mon.c (mon.2) 5 : audit 0 from='client.44207 172.21.15.78:0/1378947879' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:14:49.189 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:48 smithi078 bash[63468]: debug 2023-07-29T17:14:48.763+0000 7ff50defd640 -1 osd.0 78 log_to_monitors true 2023-07-29T17:14:49.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[58614]: cluster 2023-07-29T17:14:48.051471+0000 mgr.y (mgr.25169) 144 : cluster 0 pgmap v75: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:49.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[58614]: audit 2023-07-29T17:14:48.773348+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:14:49.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[58614]: audit 2023-07-29T17:14:48.773927+0000 mon.a (mon.0) 288 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:14:49.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[60360]: cluster 2023-07-29T17:14:48.051471+0000 mgr.y (mgr.25169) 144 : cluster 0 pgmap v75: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:49.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[60360]: audit 2023-07-29T17:14:48.773348+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:14:49.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[60360]: audit 2023-07-29T17:14:48.773927+0000 mon.a (mon.0) 288 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:14:49.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:49 smithi129 bash[53756]: cluster 2023-07-29T17:14:48.051471+0000 mgr.y (mgr.25169) 144 : cluster 0 pgmap v75: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:49.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:49 smithi129 bash[53756]: audit 2023-07-29T17:14:48.773348+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:14:49.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:49 smithi129 bash[53756]: audit 2023-07-29T17:14:48.773927+0000 mon.a (mon.0) 288 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T17:14:50.191 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:14:49 smithi078 bash[63468]: debug 2023-07-29T17:14:49.723+0000 7ff4fcb65700 -1 osd.0 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:14:50.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:49.245869+0000 mon.a (mon.0) 289 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:49.258379+0000 mon.a (mon.0) 290 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: cluster 2023-07-29T17:14:49.679575+0000 osd.1 (osd.1) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:50.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:49.683477+0000 mon.a (mon.0) 291 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T17:14:50.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: cluster 2023-07-29T17:14:49.683715+0000 mon.a (mon.0) 292 : cluster 0 osdmap e81: 8 total, 7 up, 8 in 2023-07-29T17:14:50.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:49.686039+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:14:50.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:49.688007+0000 mon.a (mon.0) 293 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:14:50.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.052726+0000 mgr.y (mgr.25169) 145 : cluster 0 pgmap v77: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:50.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:50.056315+0000 mon.a (mon.0) 294 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:50 smithi129 bash[53756]: audit 2023-07-29T17:14:50.067369+0000 mon.a (mon.0) 295 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:49.245869+0000 mon.a (mon.0) 289 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:49.258379+0000 mon.a (mon.0) 290 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: cluster 2023-07-29T17:14:49.679575+0000 osd.1 (osd.1) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:50.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:49.683477+0000 mon.a (mon.0) 291 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T17:14:50.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: cluster 2023-07-29T17:14:49.683715+0000 mon.a (mon.0) 292 : cluster 0 osdmap e81: 8 total, 7 up, 8 in 2023-07-29T17:14:50.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:49.686039+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:14:50.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:49.688007+0000 mon.a (mon.0) 293 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:14:50.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.052726+0000 mgr.y (mgr.25169) 145 : cluster 0 pgmap v77: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:50.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:50.056315+0000 mon.a (mon.0) 294 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[58614]: audit 2023-07-29T17:14:50.067369+0000 mon.a (mon.0) 295 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:49.245869+0000 mon.a (mon.0) 289 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:49.258379+0000 mon.a (mon.0) 290 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: cluster 2023-07-29T17:14:49.679575+0000 osd.1 (osd.1) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:50.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:49.683477+0000 mon.a (mon.0) 291 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T17:14:50.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: cluster 2023-07-29T17:14:49.683715+0000 mon.a (mon.0) 292 : cluster 0 osdmap e81: 8 total, 7 up, 8 in 2023-07-29T17:14:50.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:49.686039+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:14:50.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:49.688007+0000 mon.a (mon.0) 293 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:14:50.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.052726+0000 mgr.y (mgr.25169) 145 : cluster 0 pgmap v77: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:50.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:50.056315+0000 mon.a (mon.0) 294 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:50.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:50 smithi078 bash[60360]: audit 2023-07-29T17:14:50.067369+0000 mon.a (mon.0) 295 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:14:51.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:49.681124+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.450461+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.453048+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.453060+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.678642+0000 mon.a (mon.0) 296 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:14:51.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.685957+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:51.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.688529+0000 mon.a (mon.0) 297 : cluster 1 osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141] boot 2023-07-29T17:14:51.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.688574+0000 mon.a (mon.0) 298 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2023-07-29T17:14:51.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:51 smithi129 bash[53756]: audit 2023-07-29T17:14:50.688864+0000 mon.a (mon.0) 299 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:14:51.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:49.681124+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.450461+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.453048+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.453060+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.678642+0000 mon.a (mon.0) 296 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:14:51.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.685957+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:51.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.688529+0000 mon.a (mon.0) 297 : cluster 1 osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141] boot 2023-07-29T17:14:51.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.688574+0000 mon.a (mon.0) 298 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2023-07-29T17:14:51.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[58614]: audit 2023-07-29T17:14:50.688864+0000 mon.a (mon.0) 299 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:14:51.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:49.681124+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.450461+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.453048+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.453060+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e81 with expected crc 2023-07-29T17:14:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.678642+0000 mon.a (mon.0) 296 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:14:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.685957+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.688529+0000 mon.a (mon.0) 297 : cluster 1 osd.0 [v2:172.21.15.78:6802/1852304141,v1:172.21.15.78:6803/1852304141] boot 2023-07-29T17:14:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.688574+0000 mon.a (mon.0) 298 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2023-07-29T17:14:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:51 smithi078 bash[60360]: audit 2023-07-29T17:14:50.688864+0000 mon.a (mon.0) 299 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T17:14:52.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.682819+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:52.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:50.690154+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:52.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.685016+0000 osd.1 (osd.1) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.689269+0000 mon.a (mon.0) 300 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2023-07-29T17:14:52.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.689757+0000 osd.1 (osd.1) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.692237+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.692293+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.693661+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.698762+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:52 smithi129 bash[53756]: cluster 2023-07-29T17:14:52.053751+0000 mgr.y (mgr.25169) 146 : cluster 0 pgmap v80: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:52.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.682819+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:52.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:50.690154+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:52.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.685016+0000 osd.1 (osd.1) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.689269+0000 mon.a (mon.0) 300 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2023-07-29T17:14:52.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.689757+0000 osd.1 (osd.1) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.692237+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.692293+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.693661+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.698762+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[60360]: cluster 2023-07-29T17:14:52.053751+0000 mgr.y (mgr.25169) 146 : cluster 0 pgmap v80: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:52.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.682819+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:52.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:50.690154+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-29T17:14:52.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.685016+0000 osd.1 (osd.1) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.689269+0000 mon.a (mon.0) 300 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.689757+0000 osd.1 (osd.1) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.692237+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.692293+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.693661+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.698762+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:52.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:52 smithi078 bash[58614]: cluster 2023-07-29T17:14:52.053751+0000 mgr.y (mgr.25169) 146 : cluster 0 pgmap v80: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:53.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:53 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.685916+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:53 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.689704+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:53 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.693773+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:53 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.694375+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:53 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.696590+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:53 smithi129 bash[53756]: cluster 2023-07-29T17:14:51.700815+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.685916+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.689704+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.693773+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.694375+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.696590+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[58614]: cluster 2023-07-29T17:14:51.700815+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.685916+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.689704+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.693773+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.694375+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.696590+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:53.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:53 smithi078 bash[60360]: cluster 2023-07-29T17:14:51.700815+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e83 with expected crc 2023-07-29T17:14:55.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:54 smithi078 bash[60360]: cluster 2023-07-29T17:14:54.054734+0000 mgr.y (mgr.25169) 147 : cluster 0 pgmap v81: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:55.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:54 smithi078 bash[58614]: cluster 2023-07-29T17:14:54.054734+0000 mgr.y (mgr.25169) 147 : cluster 0 pgmap v81: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:55.172 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:54 smithi129 bash[53756]: cluster 2023-07-29T17:14:54.054734+0000 mgr.y (mgr.25169) 147 : cluster 0 pgmap v81: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-29T17:14:56.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:56 smithi078 bash[58614]: cluster 2023-07-29T17:14:56.108402+0000 mon.a (mon.0) 301 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded) 2023-07-29T17:14:56.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:56 smithi078 bash[60360]: cluster 2023-07-29T17:14:56.108402+0000 mon.a (mon.0) 301 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded) 2023-07-29T17:14:56.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:56 smithi129 bash[53756]: cluster 2023-07-29T17:14:56.108402+0000 mon.a (mon.0) 301 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 114/999 objects degraded (11.411%), 19 pgs degraded) 2023-07-29T17:14:57.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:57 smithi078 bash[58614]: cluster 2023-07-29T17:14:56.056418+0000 mgr.y (mgr.25169) 148 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:57.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:57 smithi078 bash[60360]: cluster 2023-07-29T17:14:56.056418+0000 mgr.y (mgr.25169) 148 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:57.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:57 smithi129 bash[53756]: cluster 2023-07-29T17:14:56.056418+0000 mgr.y (mgr.25169) 148 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:59.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:14:59 smithi078 bash[58614]: cluster 2023-07-29T17:14:58.057240+0000 mgr.y (mgr.25169) 149 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:59.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:14:59 smithi078 bash[60360]: cluster 2023-07-29T17:14:58.057240+0000 mgr.y (mgr.25169) 149 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:14:59.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:14:59 smithi129 bash[53756]: cluster 2023-07-29T17:14:58.057240+0000 mgr.y (mgr.25169) 149 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: cluster 2023-07-29T17:15:00.058442+0000 mgr.y (mgr.25169) 150 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.065555+0000 mon.a (mon.0) 302 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.075185+0000 mon.a (mon.0) 303 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.076900+0000 mon.a (mon.0) 304 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.078138+0000 mon.a (mon.0) 305 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:15:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: cephadm 2023-07-29T17:15:00.079180+0000 mgr.y (mgr.25169) 151 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:15:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: cluster 2023-07-29T17:15:00.080185+0000 mgr.y (mgr.25169) 152 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: cluster 2023-07-29T17:15:00.080492+0000 mgr.y (mgr.25169) 153 : cluster 0 pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.086883+0000 mon.a (mon.0) 306 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.178329+0000 mon.a (mon.0) 307 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:15:01.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.182041+0000 mon.a (mon.0) 308 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.184360+0000 mon.a (mon.0) 309 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.186364+0000 mon.a (mon.0) 310 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:01 smithi129 bash[53756]: audit 2023-07-29T17:15:00.188771+0000 mon.a (mon.0) 311 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-29T17:15:01.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: cluster 2023-07-29T17:15:00.058442+0000 mgr.y (mgr.25169) 150 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.065555+0000 mon.a (mon.0) 302 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.075185+0000 mon.a (mon.0) 303 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.076900+0000 mon.a (mon.0) 304 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.078138+0000 mon.a (mon.0) 305 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: cephadm 2023-07-29T17:15:00.079180+0000 mgr.y (mgr.25169) 151 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: cluster 2023-07-29T17:15:00.080185+0000 mgr.y (mgr.25169) 152 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: cluster 2023-07-29T17:15:00.080492+0000 mgr.y (mgr.25169) 153 : cluster 0 pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.086883+0000 mon.a (mon.0) 306 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.178329+0000 mon.a (mon.0) 307 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:15:01.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.182041+0000 mon.a (mon.0) 308 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.184360+0000 mon.a (mon.0) 309 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.186364+0000 mon.a (mon.0) 310 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[60360]: audit 2023-07-29T17:15:00.188771+0000 mon.a (mon.0) 311 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-29T17:15:01.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: cluster 2023-07-29T17:15:00.058442+0000 mgr.y (mgr.25169) 150 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.065555+0000 mon.a (mon.0) 302 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.075185+0000 mon.a (mon.0) 303 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.076900+0000 mon.a (mon.0) 304 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.078138+0000 mon.a (mon.0) 305 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:15:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: cephadm 2023-07-29T17:15:00.079180+0000 mgr.y (mgr.25169) 151 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:15:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: cluster 2023-07-29T17:15:00.080185+0000 mgr.y (mgr.25169) 152 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: cluster 2023-07-29T17:15:00.080492+0000 mgr.y (mgr.25169) 153 : cluster 0 pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.086883+0000 mon.a (mon.0) 306 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.178329+0000 mon.a (mon.0) 307 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.182041+0000 mon.a (mon.0) 308 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.184360+0000 mon.a (mon.0) 309 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.186364+0000 mon.a (mon.0) 310 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:01 smithi078 bash[58614]: audit 2023-07-29T17:15:00.188771+0000 mon.a (mon.0) 311 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-29T17:15:02.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:02 smithi078 bash[58614]: audit 2023-07-29T17:15:00.189045+0000 mgr.y (mgr.25169) 154 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-29T17:15:02.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:02 smithi078 bash[58614]: cephadm 2023-07-29T17:15:00.190859+0000 mgr.y (mgr.25169) 155 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-07-29T17:15:02.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:02 smithi078 bash[60360]: audit 2023-07-29T17:15:00.189045+0000 mgr.y (mgr.25169) 154 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-29T17:15:02.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:02 smithi078 bash[60360]: cephadm 2023-07-29T17:15:00.190859+0000 mgr.y (mgr.25169) 155 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-07-29T17:15:02.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:02 smithi129 bash[53756]: audit 2023-07-29T17:15:00.189045+0000 mgr.y (mgr.25169) 154 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-29T17:15:02.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:02 smithi129 bash[53756]: cephadm 2023-07-29T17:15:00.190859+0000 mgr.y (mgr.25169) 155 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-07-29T17:15:03.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[58614]: cluster 2023-07-29T17:15:02.081212+0000 mgr.y (mgr.25169) 156 : cluster 0 pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:03.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[58614]: audit 2023-07-29T17:15:02.523064+0000 mon.a (mon.0) 312 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:03.099 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[58614]: audit 2023-07-29T17:15:02.525054+0000 mon.a (mon.0) 313 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:15:03.099 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[58614]: audit 2023-07-29T17:15:02.526479+0000 mon.a (mon.0) 314 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[60360]: cluster 2023-07-29T17:15:02.081212+0000 mgr.y (mgr.25169) 156 : cluster 0 pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[60360]: audit 2023-07-29T17:15:02.523064+0000 mon.a (mon.0) 312 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[60360]: audit 2023-07-29T17:15:02.525054+0000 mon.a (mon.0) 313 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:15:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:03 smithi078 bash[60360]: audit 2023-07-29T17:15:02.526479+0000 mon.a (mon.0) 314 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:03.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:03 smithi129 bash[53756]: cluster 2023-07-29T17:15:02.081212+0000 mgr.y (mgr.25169) 156 : cluster 0 pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:03.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:03 smithi129 bash[53756]: audit 2023-07-29T17:15:02.523064+0000 mon.a (mon.0) 312 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:03.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:03 smithi129 bash[53756]: audit 2023-07-29T17:15:02.525054+0000 mon.a (mon.0) 313 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T17:15:03.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:03 smithi129 bash[53756]: audit 2023-07-29T17:15:02.526479+0000 mon.a (mon.0) 314 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:04.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:04 smithi078 bash[58614]: cephadm 2023-07-29T17:15:02.510130+0000 mgr.y (mgr.25169) 157 : cephadm 1 Upgrade: Updating osd.1 2023-07-29T17:15:04.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:04 smithi078 bash[58614]: cephadm 2023-07-29T17:15:02.531775+0000 mgr.y (mgr.25169) 158 : cephadm 1 Deploying daemon osd.1 on smithi078 2023-07-29T17:15:04.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:04 smithi078 bash[58614]: audit 2023-07-29T17:15:03.294653+0000 mon.a (mon.0) 315 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:04.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:04 smithi078 bash[60360]: cephadm 2023-07-29T17:15:02.510130+0000 mgr.y (mgr.25169) 157 : cephadm 1 Upgrade: Updating osd.1 2023-07-29T17:15:04.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:04 smithi078 bash[60360]: cephadm 2023-07-29T17:15:02.531775+0000 mgr.y (mgr.25169) 158 : cephadm 1 Deploying daemon osd.1 on smithi078 2023-07-29T17:15:04.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:04 smithi078 bash[60360]: audit 2023-07-29T17:15:03.294653+0000 mon.a (mon.0) 315 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:04.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:04 smithi129 bash[53756]: cephadm 2023-07-29T17:15:02.510130+0000 mgr.y (mgr.25169) 157 : cephadm 1 Upgrade: Updating osd.1 2023-07-29T17:15:04.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:04 smithi129 bash[53756]: cephadm 2023-07-29T17:15:02.531775+0000 mgr.y (mgr.25169) 158 : cephadm 1 Deploying daemon osd.1 on smithi078 2023-07-29T17:15:04.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:04 smithi129 bash[53756]: audit 2023-07-29T17:15:03.294653+0000 mon.a (mon.0) 315 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:05.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:05 smithi129 bash[53756]: cluster 2023-07-29T17:15:04.081804+0000 mgr.y (mgr.25169) 159 : cluster 0 pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:05.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:05 smithi078 bash[58614]: cluster 2023-07-29T17:15:04.081804+0000 mgr.y (mgr.25169) 159 : cluster 0 pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:05.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:05 smithi078 bash[60360]: cluster 2023-07-29T17:15:04.081804+0000 mgr.y (mgr.25169) 159 : cluster 0 pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:07.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:07 smithi078 bash[58614]: cluster 2023-07-29T17:15:06.083106+0000 mgr.y (mgr.25169) 160 : cluster 0 pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:07.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:07 smithi078 bash[60360]: cluster 2023-07-29T17:15:06.083106+0000 mgr.y (mgr.25169) 160 : cluster 0 pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:07.422 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:07 smithi078 systemd[1]: Stopping Ceph osd.1 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:15:07.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:07 smithi129 bash[53756]: cluster 2023-07-29T17:15:06.083106+0000 mgr.y (mgr.25169) 160 : cluster 0 pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:07.691 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:07 smithi078 bash[33691]: debug 2023-07-29T17:15:07.399+0000 7ff729e35700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:15:07.692 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:07 smithi078 bash[33691]: debug 2023-07-29T17:15:07.399+0000 7ff729e35700 -1 osd.1 83 *** Got signal Terminated *** 2023-07-29T17:15:07.692 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:07 smithi078 bash[33691]: debug 2023-07-29T17:15:07.399+0000 7ff729e35700 -1 osd.1 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:15:08.413 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[65237]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.1 2023-07-29T17:15:08.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.664601+0000 mon.a (mon.0) 316 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.664799+0000 mon.a (mon.0) 317 : cluster 1 osd.1 failed (root=default,host=smithi078) (connection refused reported by osd.3) 2023-07-29T17:15:08.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.665569+0000 mon.a (mon.0) 318 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.665978+0000 mon.a (mon.0) 319 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.666286+0000 mon.a (mon.0) 320 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.666478+0000 mon.a (mon.0) 321 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.666643+0000 mon.a (mon.0) 322 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.666858+0000 mon.a (mon.0) 323 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.667064+0000 mon.a (mon.0) 324 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.667248+0000 mon.a (mon.0) 325 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.667485+0000 mon.a (mon.0) 326 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.667898+0000 mon.a (mon.0) 327 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.668225+0000 mon.a (mon.0) 328 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.668546+0000 mon.a (mon.0) 329 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.668898+0000 mon.a (mon.0) 330 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.705224+0000 mon.a (mon.0) 331 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.709381+0000 mon.a (mon.0) 332 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.713610+0000 mon.a (mon.0) 333 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.714646+0000 mon.a (mon.0) 334 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.726631+0000 mon.a (mon.0) 335 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.908723+0000 mon.a (mon.0) 336 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.913537+0000 mon.a (mon.0) 337 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.916517+0000 mon.a (mon.0) 338 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:07.917260+0000 mon.a (mon.0) 339 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.068461+0000 mon.a (mon.0) 340 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.068875+0000 mon.a (mon.0) 341 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.664601+0000 mon.a (mon.0) 316 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.664799+0000 mon.a (mon.0) 317 : cluster 1 osd.1 failed (root=default,host=smithi078) (connection refused reported by osd.3) 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.665569+0000 mon.a (mon.0) 318 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.665978+0000 mon.a (mon.0) 319 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.666286+0000 mon.a (mon.0) 320 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.666478+0000 mon.a (mon.0) 321 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.666643+0000 mon.a (mon.0) 322 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.666858+0000 mon.a (mon.0) 323 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.667064+0000 mon.a (mon.0) 324 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.667248+0000 mon.a (mon.0) 325 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.667485+0000 mon.a (mon.0) 326 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.667898+0000 mon.a (mon.0) 327 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.668225+0000 mon.a (mon.0) 328 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.668546+0000 mon.a (mon.0) 329 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.668898+0000 mon.a (mon.0) 330 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.705224+0000 mon.a (mon.0) 331 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.709381+0000 mon.a (mon.0) 332 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.713610+0000 mon.a (mon.0) 333 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.714646+0000 mon.a (mon.0) 334 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.726631+0000 mon.a (mon.0) 335 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.908723+0000 mon.a (mon.0) 336 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.913537+0000 mon.a (mon.0) 337 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.916517+0000 mon.a (mon.0) 338 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:07.917260+0000 mon.a (mon.0) 339 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.068461+0000 mon.a (mon.0) 340 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.068875+0000 mon.a (mon.0) 341 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.069400+0000 mon.a (mon.0) 342 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.069797+0000 mon.a (mon.0) 343 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.070075+0000 mon.a (mon.0) 344 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.070266+0000 mon.a (mon.0) 345 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.070469+0000 mon.a (mon.0) 346 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.070752+0000 mon.a (mon.0) 347 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.070863+0000 mon.a (mon.0) 348 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.071188+0000 mon.a (mon.0) 349 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.071374+0000 mon.a (mon.0) 350 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.071727+0000 mon.a (mon.0) 351 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.071911+0000 mon.a (mon.0) 352 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.072228+0000 mon.a (mon.0) 353 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.069400+0000 mon.a (mon.0) 342 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.069797+0000 mon.a (mon.0) 343 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.070075+0000 mon.a (mon.0) 344 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.070266+0000 mon.a (mon.0) 345 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.070469+0000 mon.a (mon.0) 346 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.070752+0000 mon.a (mon.0) 347 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.070863+0000 mon.a (mon.0) 348 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.071188+0000 mon.a (mon.0) 349 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.071374+0000 mon.a (mon.0) 350 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.071727+0000 mon.a (mon.0) 351 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.071911+0000 mon.a (mon.0) 352 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:08 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.072228+0000 mon.a (mon.0) 353 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.664601+0000 mon.a (mon.0) 316 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.664799+0000 mon.a (mon.0) 317 : cluster 1 osd.1 failed (root=default,host=smithi078) (connection refused reported by osd.3) 2023-07-29T17:15:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.665569+0000 mon.a (mon.0) 318 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.665978+0000 mon.a (mon.0) 319 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.666286+0000 mon.a (mon.0) 320 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.666478+0000 mon.a (mon.0) 321 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.666643+0000 mon.a (mon.0) 322 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.666858+0000 mon.a (mon.0) 323 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.667064+0000 mon.a (mon.0) 324 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.667248+0000 mon.a (mon.0) 325 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.667485+0000 mon.a (mon.0) 326 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.667898+0000 mon.a (mon.0) 327 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.668225+0000 mon.a (mon.0) 328 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.668546+0000 mon.a (mon.0) 329 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.668898+0000 mon.a (mon.0) 330 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.705224+0000 mon.a (mon.0) 331 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.709381+0000 mon.a (mon.0) 332 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.713610+0000 mon.a (mon.0) 333 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.714646+0000 mon.a (mon.0) 334 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.726631+0000 mon.a (mon.0) 335 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.908723+0000 mon.a (mon.0) 336 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.913537+0000 mon.a (mon.0) 337 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.916517+0000 mon.a (mon.0) 338 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:07.917260+0000 mon.a (mon.0) 339 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.068461+0000 mon.a (mon.0) 340 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.068875+0000 mon.a (mon.0) 341 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.069400+0000 mon.a (mon.0) 342 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.069797+0000 mon.a (mon.0) 343 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-29T17:15:08.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.070075+0000 mon.a (mon.0) 344 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.070266+0000 mon.a (mon.0) 345 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.070469+0000 mon.a (mon.0) 346 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.070752+0000 mon.a (mon.0) 347 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.070863+0000 mon.a (mon.0) 348 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.071188+0000 mon.a (mon.0) 349 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.071374+0000 mon.a (mon.0) 350 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.071727+0000 mon.a (mon.0) 351 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-29T17:15:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.071911+0000 mon.a (mon.0) 352 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:08.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:08 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.072228+0000 mon.a (mon.0) 353 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-29T17:15:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.083737+0000 mgr.y (mgr.25169) 161 : cluster 0 pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.091799+0000 mon.a (mon.0) 354 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:15:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.104197+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.105657+0000 mon.a (mon.0) 355 : cluster 0 osdmap e84: 8 total, 7 up, 8 in 2023-07-29T17:15:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.106712+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.107464+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.110274+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.110295+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.112712+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:09 smithi129 bash[53756]: cluster 2023-07-29T17:15:08.112967+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.083737+0000 mgr.y (mgr.25169) 161 : cluster 0 pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:09.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.091799+0000 mon.a (mon.0) 354 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:15:09.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.104197+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.105657+0000 mon.a (mon.0) 355 : cluster 0 osdmap e84: 8 total, 7 up, 8 in 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.106712+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.107464+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.110274+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.110295+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.112712+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[58614]: cluster 2023-07-29T17:15:08.112967+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.083737+0000 mgr.y (mgr.25169) 161 : cluster 0 pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.091799+0000 mon.a (mon.0) 354 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.104197+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.105657+0000 mon.a (mon.0) 355 : cluster 0 osdmap e84: 8 total, 7 up, 8 in 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.106712+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.107464+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.110274+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.110295+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.112712+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:09 smithi078 bash[60360]: cluster 2023-07-29T17:15:08.112967+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-29T17:15:10.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.110417+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.113326+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.113989+0000 mon.a (mon.0) 356 : cluster 0 osdmap e85: 8 total, 7 up, 8 in 2023-07-29T17:15:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.114361+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.114742+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.117016+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.117044+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.117323+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:10 smithi129 bash[53756]: cluster 2023-07-29T17:15:09.119759+0000 osd.2 (osd.2) 11 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.110417+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.113326+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.113989+0000 mon.a (mon.0) 356 : cluster 0 osdmap e85: 8 total, 7 up, 8 in 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.114361+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.114742+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.117016+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.117044+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.117323+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[58614]: cluster 2023-07-29T17:15:09.119759+0000 osd.2 (osd.2) 11 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.110417+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.113326+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.113989+0000 mon.a (mon.0) 356 : cluster 0 osdmap e85: 8 total, 7 up, 8 in 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.114361+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.114742+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.117016+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.117044+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.117323+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:10 smithi078 bash[60360]: cluster 2023-07-29T17:15:09.119759+0000 osd.2 (osd.2) 11 : cluster 3 failed to encode map e85 with expected crc 2023-07-29T17:15:10.941 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:10 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.1.service: Succeeded. 2023-07-29T17:15:10.942 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:10 smithi078 systemd[1]: Stopped Ceph osd.1 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:15:10.942 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:10 smithi078 systemd[1]: Started Ceph osd.1 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:15:11.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:11 smithi129 bash[53756]: cluster 2023-07-29T17:15:10.085698+0000 mgr.y (mgr.25169) 162 : cluster 0 pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:11.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:11 smithi129 bash[53756]: cluster 2023-07-29T17:15:10.109549+0000 mon.a (mon.0) 357 : cluster 3 Health check failed: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:11.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:11 smithi129 bash[53756]: audit 2023-07-29T17:15:10.930120+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:11.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:11 smithi129 bash[53756]: audit 2023-07-29T17:15:10.941192+0000 mon.a (mon.0) 359 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:11.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[58614]: cluster 2023-07-29T17:15:10.085698+0000 mgr.y (mgr.25169) 162 : cluster 0 pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:11.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[58614]: cluster 2023-07-29T17:15:10.109549+0000 mon.a (mon.0) 357 : cluster 3 Health check failed: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:11.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[58614]: audit 2023-07-29T17:15:10.930120+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:11.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[58614]: audit 2023-07-29T17:15:10.941192+0000 mon.a (mon.0) 359 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:11.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[60360]: cluster 2023-07-29T17:15:10.085698+0000 mgr.y (mgr.25169) 162 : cluster 0 pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:11.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[60360]: cluster 2023-07-29T17:15:10.109549+0000 mon.a (mon.0) 357 : cluster 3 Health check failed: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:11.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[60360]: audit 2023-07-29T17:15:10.930120+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:11.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:11 smithi078 bash[60360]: audit 2023-07-29T17:15:10.941192+0000 mon.a (mon.0) 359 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:13.148 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-29T17:15:13.148 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-07-29T17:15:13.149 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-07-29T17:15:13.149 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-29T17:15:13.149 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2023-07-29T17:15:13.149 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-29T17:15:13.149 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:12 smithi078 bash[65509]: --> ceph-volume raw activate successful for osd ID: 1 2023-07-29T17:15:13.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:13 smithi129 bash[53756]: cluster 2023-07-29T17:15:12.086904+0000 mgr.y (mgr.25169) 163 : cluster 0 pgmap v94: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:13.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:13 smithi078 bash[58614]: cluster 2023-07-29T17:15:12.086904+0000 mgr.y (mgr.25169) 163 : cluster 0 pgmap v94: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:13.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:13 smithi078 bash[60360]: cluster 2023-07-29T17:15:12.086904+0000 mgr.y (mgr.25169) 163 : cluster 0 pgmap v94: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:14.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:14 smithi078 bash[58614]: cluster 2023-07-29T17:15:14.087950+0000 mgr.y (mgr.25169) 164 : cluster 0 pgmap v95: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:14.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:14 smithi078 bash[60360]: cluster 2023-07-29T17:15:14.087950+0000 mgr.y (mgr.25169) 164 : cluster 0 pgmap v95: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:14.943 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:14 smithi078 bash[65820]: debug 2023-07-29T17:15:14.639+0000 7fc895980700 1 -- 172.21.15.78:0/1069171564 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5609517784e0 con 0x5609517d3000 2023-07-29T17:15:15.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:14 smithi129 bash[53756]: cluster 2023-07-29T17:15:14.087950+0000 mgr.y (mgr.25169) 164 : cluster 0 pgmap v95: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:15.692 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:15 smithi078 bash[65820]: debug 2023-07-29T17:15:15.179+0000 7fc89eb05640 -1 Falling back to public interface 2023-07-29T17:15:17.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:17 smithi129 bash[53756]: cluster 2023-07-29T17:15:16.089903+0000 mgr.y (mgr.25169) 165 : cluster 0 pgmap v96: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:17.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:17 smithi078 bash[58614]: cluster 2023-07-29T17:15:16.089903+0000 mgr.y (mgr.25169) 165 : cluster 0 pgmap v96: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:17.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:17 smithi078 bash[60360]: cluster 2023-07-29T17:15:16.089903+0000 mgr.y (mgr.25169) 165 : cluster 0 pgmap v96: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:17.741 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:15:18.569 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:15:18.570 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5m) 0s ago 11m 10.2M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:15:18.570 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (4m) 97s ago 11m 26.1M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:15:18.570 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 97s ago 9m - - 2023-07-29T17:15:18.570 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (3m) 97s ago 15m 428M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:15:18.570 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (4m) 0s ago 17m 471M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:15:18.570 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2m) 0s ago 17m 57.8M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (106s) 97s ago 16m 37.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2m) 0s ago 16m 47.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5m) 0s ago 11m 9339k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (4m) 97s ago 11m 3644k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (34s) 0s ago 15m 48.1M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4s) 0s ago 14m - 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:15:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (14m) 0s ago 14m 50.7M 4096M 16.2.0 24ecd6d5f14c 35221f289608 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (14m) 0s ago 14m 56.1M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (13m) 97s ago 13m 55.5M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (13m) 97s ago 13m 53.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (12m) 97s ago 12m 53.3M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (12m) 97s ago 12m 56.5M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 97s ago 11m - - 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (9m) 0s ago 9m 55.2M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:15:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (9m) 97s ago 9m 54.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:15:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (9m) 0s ago 9m 55.3M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:15:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (9m) 97s ago 9m 54.5M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:15:18.692 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:18 smithi078 bash[65820]: debug 2023-07-29T17:15:18.223+0000 7fc89eb05640 -1 osd.1 83 log_to_monitors true 2023-07-29T17:15:19.000 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:15:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:15:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:15:19.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:19.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:15:19.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:15:19.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 1 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:19.002 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:15:19.003 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 10, 2023-07-29T17:15:19.003 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 6 2023-07-29T17:15:19.003 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:15:19.003 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:15:19.393 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:15:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:15:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:15:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:15:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:15:19.393 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:15:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:15:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:15:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "7/23 daemons upgraded", 2023-07-29T17:15:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:15:19.394 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:15:19.394 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:15:19.442 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[65820]: debug 2023-07-29T17:15:19.203+0000 7fc88d76d700 -1 osd.1 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:15:19.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: audit 2023-07-29T17:15:17.704963+0000 mgr.y (mgr.25169) 166 : audit 0 from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:19.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: cluster 2023-07-29T17:15:18.090637+0000 mgr.y (mgr.25169) 167 : cluster 0 pgmap v97: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:19.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: audit 2023-07-29T17:15:18.110083+0000 mgr.y (mgr.25169) 168 : audit 0 from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:19.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: audit 2023-07-29T17:15:18.230272+0000 mon.a (mon.0) 360 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:15:19.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: audit 2023-07-29T17:15:18.506186+0000 mon.a (mon.0) 361 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:19.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: audit 2023-07-29T17:15:18.513813+0000 mon.a (mon.0) 362 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:19.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[58614]: audit 2023-07-29T17:15:18.978060+0000 mon.c (mon.2) 8 : audit 0 from='client.44240 172.21.15.78:0/282451979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:19.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: audit 2023-07-29T17:15:17.704963+0000 mgr.y (mgr.25169) 166 : audit 0 from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:19.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: cluster 2023-07-29T17:15:18.090637+0000 mgr.y (mgr.25169) 167 : cluster 0 pgmap v97: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:19.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: audit 2023-07-29T17:15:18.110083+0000 mgr.y (mgr.25169) 168 : audit 0 from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:19.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: audit 2023-07-29T17:15:18.230272+0000 mon.a (mon.0) 360 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:15:19.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: audit 2023-07-29T17:15:18.506186+0000 mon.a (mon.0) 361 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:19.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: audit 2023-07-29T17:15:18.513813+0000 mon.a (mon.0) 362 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:19.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:19 smithi078 bash[60360]: audit 2023-07-29T17:15:18.978060+0000 mon.c (mon.2) 8 : audit 0 from='client.44240 172.21.15.78:0/282451979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:19.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: audit 2023-07-29T17:15:17.704963+0000 mgr.y (mgr.25169) 166 : audit 0 from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:19.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: cluster 2023-07-29T17:15:18.090637+0000 mgr.y (mgr.25169) 167 : cluster 0 pgmap v97: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:19.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: audit 2023-07-29T17:15:18.110083+0000 mgr.y (mgr.25169) 168 : audit 0 from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: audit 2023-07-29T17:15:18.230272+0000 mon.a (mon.0) 360 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T17:15:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: audit 2023-07-29T17:15:18.506186+0000 mon.a (mon.0) 361 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: audit 2023-07-29T17:15:18.513813+0000 mon.a (mon.0) 362 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:19 smithi129 bash[53756]: audit 2023-07-29T17:15:18.978060+0000 mon.c (mon.2) 8 : audit 0 from='client.44240 172.21.15.78:0/282451979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:19.833 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded 2023-07-29T17:15:19.834 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:15:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:15:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:15:19.834 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] OSD_DOWN: 1 osds down 2023-07-29T17:15:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: osd.1 (root=default,host=smithi078) is down 2023-07-29T17:15:19.834 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2023-07-29T17:15:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2023-07-29T17:15:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2023-07-29T17:15:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2023-07-29T17:15:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2023-07-29T17:15:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2023-07-29T17:15:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2023-07-29T17:15:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2023-07-29T17:15:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2023-07-29T17:15:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: audit 2023-07-29T17:15:18.524803+0000 mgr.y (mgr.25169) 169 : audit 0 from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:20.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: cluster 2023-07-29T17:15:19.152498+0000 osd.2 (osd.2) 12 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: audit 2023-07-29T17:15:19.153735+0000 mon.a (mon.0) 363 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T17:15:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: cluster 2023-07-29T17:15:19.153925+0000 mon.a (mon.0) 364 : cluster 0 osdmap e86: 8 total, 7 up, 8 in 2023-07-29T17:15:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: cluster 2023-07-29T17:15:19.154177+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: audit 2023-07-29T17:15:19.155150+0000 mon.a (mon.0) 365 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:15:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: audit 2023-07-29T17:15:19.311094+0000 mon.a (mon.0) 366 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: audit 2023-07-29T17:15:19.320645+0000 mon.a (mon.0) 367 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: audit 2023-07-29T17:15:19.812479+0000 mon.a (mon.0) 368 : audit 0 from='client.44252 172.21.15.78:0/424997977' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:15:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:20 smithi129 bash[53756]: cluster 2023-07-29T17:15:19.896962+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: audit 2023-07-29T17:15:18.524803+0000 mgr.y (mgr.25169) 169 : audit 0 from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: cluster 2023-07-29T17:15:19.152498+0000 osd.2 (osd.2) 12 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: audit 2023-07-29T17:15:19.153735+0000 mon.a (mon.0) 363 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T17:15:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: cluster 2023-07-29T17:15:19.153925+0000 mon.a (mon.0) 364 : cluster 0 osdmap e86: 8 total, 7 up, 8 in 2023-07-29T17:15:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: cluster 2023-07-29T17:15:19.154177+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: audit 2023-07-29T17:15:19.155150+0000 mon.a (mon.0) 365 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:15:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: audit 2023-07-29T17:15:19.311094+0000 mon.a (mon.0) 366 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: audit 2023-07-29T17:15:19.320645+0000 mon.a (mon.0) 367 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: audit 2023-07-29T17:15:19.812479+0000 mon.a (mon.0) 368 : audit 0 from='client.44252 172.21.15.78:0/424997977' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:15:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[58614]: cluster 2023-07-29T17:15:19.896962+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: audit 2023-07-29T17:15:18.524803+0000 mgr.y (mgr.25169) 169 : audit 0 from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:20.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: cluster 2023-07-29T17:15:19.152498+0000 osd.2 (osd.2) 12 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: audit 2023-07-29T17:15:19.153735+0000 mon.a (mon.0) 363 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T17:15:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: cluster 2023-07-29T17:15:19.153925+0000 mon.a (mon.0) 364 : cluster 0 osdmap e86: 8 total, 7 up, 8 in 2023-07-29T17:15:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: cluster 2023-07-29T17:15:19.154177+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: audit 2023-07-29T17:15:19.155150+0000 mon.a (mon.0) 365 : audit 1 from='osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:15:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: audit 2023-07-29T17:15:19.311094+0000 mon.a (mon.0) 366 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: audit 2023-07-29T17:15:19.320645+0000 mon.a (mon.0) 367 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: audit 2023-07-29T17:15:19.812479+0000 mon.a (mon.0) 368 : audit 0 from='client.44252 172.21.15.78:0/424997977' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:15:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:20 smithi078 bash[60360]: cluster 2023-07-29T17:15:19.896962+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e86 with expected crc 2023-07-29T17:15:21.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: audit 2023-07-29T17:15:19.371593+0000 mgr.y (mgr.25169) 170 : audit 0 from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:21.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: cluster 2023-07-29T17:15:20.092724+0000 mgr.y (mgr.25169) 171 : cluster 0 pgmap v99: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:21.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: cluster 2023-07-29T17:15:20.160562+0000 mon.a (mon.0) 369 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:15:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: cluster 2023-07-29T17:15:20.166160+0000 osd.2 (osd.2) 13 : cluster 3 failed to encode map e87 with expected crc 2023-07-29T17:15:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: cluster 2023-07-29T17:15:20.169233+0000 mon.a (mon.0) 370 : cluster 1 osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190] boot 2023-07-29T17:15:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: cluster 2023-07-29T17:15:20.169349+0000 mon.a (mon.0) 371 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2023-07-29T17:15:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: audit 2023-07-29T17:15:20.170472+0000 mon.a (mon.0) 372 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:15:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:21 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.173479+0000 mon.a (mon.0) 373 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2023-07-29T17:15:21.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: audit 2023-07-29T17:15:19.371593+0000 mgr.y (mgr.25169) 170 : audit 0 from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:21.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: cluster 2023-07-29T17:15:20.092724+0000 mgr.y (mgr.25169) 171 : cluster 0 pgmap v99: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:21.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: cluster 2023-07-29T17:15:20.160562+0000 mon.a (mon.0) 369 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:15:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: cluster 2023-07-29T17:15:20.166160+0000 osd.2 (osd.2) 13 : cluster 3 failed to encode map e87 with expected crc 2023-07-29T17:15:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: cluster 2023-07-29T17:15:20.169233+0000 mon.a (mon.0) 370 : cluster 1 osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190] boot 2023-07-29T17:15:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: cluster 2023-07-29T17:15:20.169349+0000 mon.a (mon.0) 371 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2023-07-29T17:15:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: audit 2023-07-29T17:15:20.170472+0000 mon.a (mon.0) 372 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:15:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.173479+0000 mon.a (mon.0) 373 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2023-07-29T17:15:21.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: audit 2023-07-29T17:15:19.371593+0000 mgr.y (mgr.25169) 170 : audit 0 from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: cluster 2023-07-29T17:15:20.092724+0000 mgr.y (mgr.25169) 171 : cluster 0 pgmap v99: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: cluster 2023-07-29T17:15:20.160562+0000 mon.a (mon.0) 369 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: cluster 2023-07-29T17:15:20.166160+0000 osd.2 (osd.2) 13 : cluster 3 failed to encode map e87 with expected crc 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: cluster 2023-07-29T17:15:20.169233+0000 mon.a (mon.0) 370 : cluster 1 osd.1 [v2:172.21.15.78:6810/3359428190,v1:172.21.15.78:6811/3359428190] boot 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: cluster 2023-07-29T17:15:20.169349+0000 mon.a (mon.0) 371 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: audit 2023-07-29T17:15:20.170472+0000 mon.a (mon.0) 372 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T17:15:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:21 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.173479+0000 mon.a (mon.0) 373 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2023-07-29T17:15:22.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.171620+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.174130+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.174310+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.176501+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.176931+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.180351+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.182866+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.184870+0000 osd.2 (osd.2) 14 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.187227+0000 osd.2 (osd.2) 15 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:21.189138+0000 osd.2 (osd.2) 16 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:22 smithi129 bash[53756]: cluster 2023-07-29T17:15:22.093805+0000 mgr.y (mgr.25169) 172 : cluster 0 pgmap v102: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:22.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.171620+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.174130+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.174310+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.176501+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.176931+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.180351+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.182866+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.184870+0000 osd.2 (osd.2) 14 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.187227+0000 osd.2 (osd.2) 15 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:21.189138+0000 osd.2 (osd.2) 16 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[58614]: cluster 2023-07-29T17:15:22.093805+0000 mgr.y (mgr.25169) 172 : cluster 0 pgmap v102: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:22.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.171620+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.174130+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.174310+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.176501+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.176931+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.180351+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.182866+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.184870+0000 osd.2 (osd.2) 14 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.187227+0000 osd.2 (osd.2) 15 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:21.189138+0000 osd.2 (osd.2) 16 : cluster 3 failed to encode map e88 with expected crc 2023-07-29T17:15:22.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:22 smithi078 bash[60360]: cluster 2023-07-29T17:15:22.093805+0000 mgr.y (mgr.25169) 172 : cluster 0 pgmap v102: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:25.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:24 smithi129 bash[53756]: cluster 2023-07-29T17:15:24.094890+0000 mgr.y (mgr.25169) 173 : cluster 0 pgmap v103: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:25.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:24 smithi078 bash[58614]: cluster 2023-07-29T17:15:24.094890+0000 mgr.y (mgr.25169) 173 : cluster 0 pgmap v103: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:25.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:24 smithi078 bash[60360]: cluster 2023-07-29T17:15:24.094890+0000 mgr.y (mgr.25169) 173 : cluster 0 pgmap v103: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2023-07-29T17:15:26.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:26 smithi129 bash[53756]: cluster 2023-07-29T17:15:26.096632+0000 mgr.y (mgr.25169) 174 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:26.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:26 smithi129 bash[53756]: cluster 2023-07-29T17:15:26.148651+0000 mon.a (mon.0) 374 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded) 2023-07-29T17:15:26.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:26 smithi078 bash[58614]: cluster 2023-07-29T17:15:26.096632+0000 mgr.y (mgr.25169) 174 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:26.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:26 smithi078 bash[58614]: cluster 2023-07-29T17:15:26.148651+0000 mon.a (mon.0) 374 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded) 2023-07-29T17:15:26.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:26 smithi078 bash[60360]: cluster 2023-07-29T17:15:26.096632+0000 mgr.y (mgr.25169) 174 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:26.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:26 smithi078 bash[60360]: cluster 2023-07-29T17:15:26.148651+0000 mon.a (mon.0) 374 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 129/999 objects degraded (12.913%), 21 pgs degraded) 2023-07-29T17:15:29.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:29 smithi078 bash[58614]: cluster 2023-07-29T17:15:28.097452+0000 mgr.y (mgr.25169) 175 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:29.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:29 smithi078 bash[60360]: cluster 2023-07-29T17:15:28.097452+0000 mgr.y (mgr.25169) 175 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:29.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:29 smithi129 bash[53756]: cluster 2023-07-29T17:15:28.097452+0000 mgr.y (mgr.25169) 175 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.472851+0000 mon.a (mon.0) 375 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.484326+0000 mon.a (mon.0) 376 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.485861+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:30.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.487072+0000 mon.a (mon.0) 378 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:15:30.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: cephadm 2023-07-29T17:15:29.488472+0000 mgr.y (mgr.25169) 176 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:15:30.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: cluster 2023-07-29T17:15:29.489932+0000 mgr.y (mgr.25169) 177 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: cluster 2023-07-29T17:15:29.490241+0000 mgr.y (mgr.25169) 178 : cluster 0 pgmap v107: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.497963+0000 mon.a (mon.0) 379 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.582660+0000 mon.a (mon.0) 380 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.588124+0000 mon.a (mon.0) 381 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.590797+0000 mon.a (mon.0) 382 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.592878+0000 mon.a (mon.0) 383 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.595283+0000 mon.a (mon.0) 384 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-29T17:15:30.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: audit 2023-07-29T17:15:29.595688+0000 mgr.y (mgr.25169) 179 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-29T17:15:30.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[58614]: cephadm 2023-07-29T17:15:29.597687+0000 mgr.y (mgr.25169) 180 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-07-29T17:15:30.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.472851+0000 mon.a (mon.0) 375 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.484326+0000 mon.a (mon.0) 376 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.485861+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:30.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.487072+0000 mon.a (mon.0) 378 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:15:30.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: cephadm 2023-07-29T17:15:29.488472+0000 mgr.y (mgr.25169) 176 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: cluster 2023-07-29T17:15:29.489932+0000 mgr.y (mgr.25169) 177 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: cluster 2023-07-29T17:15:29.490241+0000 mgr.y (mgr.25169) 178 : cluster 0 pgmap v107: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.497963+0000 mon.a (mon.0) 379 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.582660+0000 mon.a (mon.0) 380 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.588124+0000 mon.a (mon.0) 381 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.590797+0000 mon.a (mon.0) 382 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.592878+0000 mon.a (mon.0) 383 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.595283+0000 mon.a (mon.0) 384 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-29T17:15:30.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: audit 2023-07-29T17:15:29.595688+0000 mgr.y (mgr.25169) 179 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-29T17:15:30.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:30 smithi078 bash[60360]: cephadm 2023-07-29T17:15:29.597687+0000 mgr.y (mgr.25169) 180 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-07-29T17:15:30.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.472851+0000 mon.a (mon.0) 375 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.484326+0000 mon.a (mon.0) 376 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.485861+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.487072+0000 mon.a (mon.0) 378 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: cephadm 2023-07-29T17:15:29.488472+0000 mgr.y (mgr.25169) 176 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: cluster 2023-07-29T17:15:29.489932+0000 mgr.y (mgr.25169) 177 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: cluster 2023-07-29T17:15:29.490241+0000 mgr.y (mgr.25169) 178 : cluster 0 pgmap v107: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.497963+0000 mon.a (mon.0) 379 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:30.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.582660+0000 mon.a (mon.0) 380 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:15:30.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.588124+0000 mon.a (mon.0) 381 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.590797+0000 mon.a (mon.0) 382 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.592878+0000 mon.a (mon.0) 383 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:30.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.595283+0000 mon.a (mon.0) 384 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-29T17:15:30.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: audit 2023-07-29T17:15:29.595688+0000 mgr.y (mgr.25169) 179 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-29T17:15:30.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:30 smithi129 bash[53756]: cephadm 2023-07-29T17:15:29.597687+0000 mgr.y (mgr.25169) 180 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-07-29T17:15:33.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[58614]: cluster 2023-07-29T17:15:31.491128+0000 mgr.y (mgr.25169) 181 : cluster 0 pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:33.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[58614]: cephadm 2023-07-29T17:15:31.917584+0000 mgr.y (mgr.25169) 182 : cephadm 1 Upgrade: Updating osd.2 2023-07-29T17:15:33.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[58614]: audit 2023-07-29T17:15:31.927889+0000 mon.a (mon.0) 385 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:33.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[58614]: audit 2023-07-29T17:15:31.930159+0000 mon.a (mon.0) 386 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:15:33.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[58614]: audit 2023-07-29T17:15:31.931718+0000 mon.a (mon.0) 387 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:33.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[58614]: cephadm 2023-07-29T17:15:31.934804+0000 mgr.y (mgr.25169) 183 : cephadm 1 Deploying daemon osd.2 on smithi078 2023-07-29T17:15:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[60360]: cluster 2023-07-29T17:15:31.491128+0000 mgr.y (mgr.25169) 181 : cluster 0 pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[60360]: cephadm 2023-07-29T17:15:31.917584+0000 mgr.y (mgr.25169) 182 : cephadm 1 Upgrade: Updating osd.2 2023-07-29T17:15:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[60360]: audit 2023-07-29T17:15:31.927889+0000 mon.a (mon.0) 385 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[60360]: audit 2023-07-29T17:15:31.930159+0000 mon.a (mon.0) 386 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:15:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[60360]: audit 2023-07-29T17:15:31.931718+0000 mon.a (mon.0) 387 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:33.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:32 smithi078 bash[60360]: cephadm 2023-07-29T17:15:31.934804+0000 mgr.y (mgr.25169) 183 : cephadm 1 Deploying daemon osd.2 on smithi078 2023-07-29T17:15:33.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:32 smithi129 bash[53756]: cluster 2023-07-29T17:15:31.491128+0000 mgr.y (mgr.25169) 181 : cluster 0 pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:32 smithi129 bash[53756]: cephadm 2023-07-29T17:15:31.917584+0000 mgr.y (mgr.25169) 182 : cephadm 1 Upgrade: Updating osd.2 2023-07-29T17:15:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:32 smithi129 bash[53756]: audit 2023-07-29T17:15:31.927889+0000 mon.a (mon.0) 385 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:32 smithi129 bash[53756]: audit 2023-07-29T17:15:31.930159+0000 mon.a (mon.0) 386 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T17:15:33.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:32 smithi129 bash[53756]: audit 2023-07-29T17:15:31.931718+0000 mon.a (mon.0) 387 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:15:33.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:32 smithi129 bash[53756]: cephadm 2023-07-29T17:15:31.934804+0000 mgr.y (mgr.25169) 183 : cephadm 1 Deploying daemon osd.2 on smithi078 2023-07-29T17:15:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:34 smithi129 bash[53756]: audit 2023-07-29T17:15:33.325728+0000 mon.a (mon.0) 388 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:34.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:34 smithi129 bash[53756]: cluster 2023-07-29T17:15:33.491950+0000 mgr.y (mgr.25169) 184 : cluster 0 pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:34.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:34 smithi078 bash[58614]: audit 2023-07-29T17:15:33.325728+0000 mon.a (mon.0) 388 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:34.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:34 smithi078 bash[58614]: cluster 2023-07-29T17:15:33.491950+0000 mgr.y (mgr.25169) 184 : cluster 0 pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:34.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:34 smithi078 bash[60360]: audit 2023-07-29T17:15:33.325728+0000 mon.a (mon.0) 388 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:34.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:34 smithi078 bash[60360]: cluster 2023-07-29T17:15:33.491950+0000 mgr.y (mgr.25169) 184 : cluster 0 pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:36.571 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:36 smithi078 systemd[1]: Stopping Ceph osd.2 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:15:36.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:36 smithi078 bash[60360]: cluster 2023-07-29T17:15:35.493630+0000 mgr.y (mgr.25169) 185 : cluster 0 pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:36.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:36 smithi078 bash[58614]: cluster 2023-07-29T17:15:35.493630+0000 mgr.y (mgr.25169) 185 : cluster 0 pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:36.850 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:36 smithi078 bash[36468]: debug 2023-07-29T17:15:36.602+0000 7fc76358f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:15:36.850 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:36 smithi078 bash[36468]: debug 2023-07-29T17:15:36.602+0000 7fc76358f700 -1 osd.2 88 *** Got signal Terminated *** 2023-07-29T17:15:36.850 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:36 smithi078 bash[36468]: debug 2023-07-29T17:15:36.602+0000 7fc76358f700 -1 osd.2 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:15:36.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:36 smithi129 bash[53756]: cluster 2023-07-29T17:15:35.493630+0000 mgr.y (mgr.25169) 185 : cluster 0 pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:37.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.655749+0000 mon.a (mon.0) 389 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.655887+0000 mon.a (mon.0) 390 : cluster 1 osd.2 failed (root=default,host=smithi078) (connection refused reported by osd.3) 2023-07-29T17:15:37.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.656484+0000 mon.a (mon.0) 391 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.657036+0000 mon.a (mon.0) 392 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.657456+0000 mon.a (mon.0) 393 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.657851+0000 mon.a (mon.0) 394 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.658219+0000 mon.a (mon.0) 395 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.658518+0000 mon.a (mon.0) 396 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.856410+0000 mon.a (mon.0) 397 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.857060+0000 mon.a (mon.0) 398 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.857484+0000 mon.a (mon.0) 399 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.857809+0000 mon.a (mon.0) 400 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.858175+0000 mon.a (mon.0) 401 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.858453+0000 mon.a (mon.0) 402 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.858703+0000 mon.a (mon.0) 403 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.858880+0000 mon.a (mon.0) 404 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.859047+0000 mon.a (mon.0) 405 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.859287+0000 mon.a (mon.0) 406 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.859635+0000 mon.a (mon.0) 407 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.860021+0000 mon.a (mon.0) 408 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.860481+0000 mon.a (mon.0) 409 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:36.861643+0000 mon.a (mon.0) 410 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.202066+0000 mon.a (mon.0) 411 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.221071+0000 mon.a (mon.0) 412 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.258415+0000 mon.a (mon.0) 413 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.260187+0000 mon.a (mon.0) 414 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.260448+0000 mon.a (mon.0) 415 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.260737+0000 mon.a (mon.0) 416 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.261262+0000 mon.a (mon.0) 417 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.261621+0000 mon.a (mon.0) 418 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.262239+0000 mon.a (mon.0) 419 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.262551+0000 mon.a (mon.0) 420 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.262810+0000 mon.a (mon.0) 421 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.263244+0000 mon.a (mon.0) 422 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.263546+0000 mon.a (mon.0) 423 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.263915+0000 mon.a (mon.0) 424 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.264343+0000 mon.a (mon.0) 425 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.264667+0000 mon.a (mon.0) 426 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.264969+0000 mon.a (mon.0) 427 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.265303+0000 mon.a (mon.0) 428 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.265641+0000 mon.a (mon.0) 429 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.404085+0000 mon.a (mon.0) 430 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.458485+0000 mon.a (mon.0) 431 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.465205+0000 mon.a (mon.0) 432 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:37 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.465452+0000 mon.a (mon.0) 433 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.942 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[67547]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.2 2023-07-29T17:15:37.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.655749+0000 mon.a (mon.0) 389 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.655887+0000 mon.a (mon.0) 390 : cluster 1 osd.2 failed (root=default,host=smithi078) (connection refused reported by osd.3) 2023-07-29T17:15:37.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.656484+0000 mon.a (mon.0) 391 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.657036+0000 mon.a (mon.0) 392 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.657456+0000 mon.a (mon.0) 393 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.657851+0000 mon.a (mon.0) 394 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.658219+0000 mon.a (mon.0) 395 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.658518+0000 mon.a (mon.0) 396 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.856410+0000 mon.a (mon.0) 397 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.857060+0000 mon.a (mon.0) 398 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.857484+0000 mon.a (mon.0) 399 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.857809+0000 mon.a (mon.0) 400 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.858175+0000 mon.a (mon.0) 401 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.858453+0000 mon.a (mon.0) 402 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.858703+0000 mon.a (mon.0) 403 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.858880+0000 mon.a (mon.0) 404 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.859047+0000 mon.a (mon.0) 405 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.859287+0000 mon.a (mon.0) 406 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.859635+0000 mon.a (mon.0) 407 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.860021+0000 mon.a (mon.0) 408 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.860481+0000 mon.a (mon.0) 409 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:36.861643+0000 mon.a (mon.0) 410 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.202066+0000 mon.a (mon.0) 411 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.221071+0000 mon.a (mon.0) 412 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.258415+0000 mon.a (mon.0) 413 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.260187+0000 mon.a (mon.0) 414 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.655749+0000 mon.a (mon.0) 389 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.655887+0000 mon.a (mon.0) 390 : cluster 1 osd.2 failed (root=default,host=smithi078) (connection refused reported by osd.3) 2023-07-29T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.656484+0000 mon.a (mon.0) 391 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.657036+0000 mon.a (mon.0) 392 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.657456+0000 mon.a (mon.0) 393 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.657851+0000 mon.a (mon.0) 394 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.658219+0000 mon.a (mon.0) 395 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.658518+0000 mon.a (mon.0) 396 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.856410+0000 mon.a (mon.0) 397 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.857060+0000 mon.a (mon.0) 398 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.857484+0000 mon.a (mon.0) 399 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.857809+0000 mon.a (mon.0) 400 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.858175+0000 mon.a (mon.0) 401 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.858453+0000 mon.a (mon.0) 402 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.858703+0000 mon.a (mon.0) 403 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.858880+0000 mon.a (mon.0) 404 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.859047+0000 mon.a (mon.0) 405 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.859287+0000 mon.a (mon.0) 406 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.859635+0000 mon.a (mon.0) 407 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.860021+0000 mon.a (mon.0) 408 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.860481+0000 mon.a (mon.0) 409 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:36.861643+0000 mon.a (mon.0) 410 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.202066+0000 mon.a (mon.0) 411 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.221071+0000 mon.a (mon.0) 412 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.258415+0000 mon.a (mon.0) 413 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.260187+0000 mon.a (mon.0) 414 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.260448+0000 mon.a (mon.0) 415 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.260737+0000 mon.a (mon.0) 416 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.261262+0000 mon.a (mon.0) 417 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.261621+0000 mon.a (mon.0) 418 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.262239+0000 mon.a (mon.0) 419 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.262551+0000 mon.a (mon.0) 420 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.262810+0000 mon.a (mon.0) 421 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.263244+0000 mon.a (mon.0) 422 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.263546+0000 mon.a (mon.0) 423 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.263915+0000 mon.a (mon.0) 424 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.264343+0000 mon.a (mon.0) 425 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.264667+0000 mon.a (mon.0) 426 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.264969+0000 mon.a (mon.0) 427 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.265303+0000 mon.a (mon.0) 428 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.265641+0000 mon.a (mon.0) 429 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.404085+0000 mon.a (mon.0) 430 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.458485+0000 mon.a (mon.0) 431 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.465205+0000 mon.a (mon.0) 432 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.465452+0000 mon.a (mon.0) 433 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.260448+0000 mon.a (mon.0) 415 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.260737+0000 mon.a (mon.0) 416 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.261262+0000 mon.a (mon.0) 417 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.261621+0000 mon.a (mon.0) 418 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.262239+0000 mon.a (mon.0) 419 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.262551+0000 mon.a (mon.0) 420 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.262810+0000 mon.a (mon.0) 421 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.263244+0000 mon.a (mon.0) 422 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.263546+0000 mon.a (mon.0) 423 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.263915+0000 mon.a (mon.0) 424 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.264343+0000 mon.a (mon.0) 425 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.264667+0000 mon.a (mon.0) 426 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.264969+0000 mon.a (mon.0) 427 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.265303+0000 mon.a (mon.0) 428 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.265641+0000 mon.a (mon.0) 429 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-29T17:15:37.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.404085+0000 mon.a (mon.0) 430 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-29T17:15:37.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.458485+0000 mon.a (mon.0) 431 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-29T17:15:37.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.465205+0000 mon.a (mon.0) 432 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-29T17:15:37.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:37 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.465452+0000 mon.a (mon.0) 433 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-29T17:15:38.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.494363+0000 mgr.y (mgr.25169) 186 : cluster 0 pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:38.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.551027+0000 mon.a (mon.0) 434 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:15:38.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.560155+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.564367+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.565011+0000 mon.a (mon.0) 435 : cluster 0 osdmap e89: 8 total, 7 up, 8 in 2023-07-29T17:15:38.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[58614]: cluster 2023-07-29T17:15:37.569893+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.494363+0000 mgr.y (mgr.25169) 186 : cluster 0 pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:38.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.551027+0000 mon.a (mon.0) 434 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:15:38.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.560155+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.564367+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.565011+0000 mon.a (mon.0) 435 : cluster 0 osdmap e89: 8 total, 7 up, 8 in 2023-07-29T17:15:38.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:38 smithi078 bash[60360]: cluster 2023-07-29T17:15:37.569893+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:38 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.494363+0000 mgr.y (mgr.25169) 186 : cluster 0 pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:38.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:38 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.551027+0000 mon.a (mon.0) 434 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:15:38.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:38 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.560155+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:38 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.564367+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:38.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:38 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.565011+0000 mon.a (mon.0) 435 : cluster 0 osdmap e89: 8 total, 7 up, 8 in 2023-07-29T17:15:38.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:38 smithi129 bash[53756]: cluster 2023-07-29T17:15:37.569893+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e89 with expected crc 2023-07-29T17:15:39.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.571246+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.573936+0000 mon.a (mon.0) 436 : cluster 0 osdmap e90: 8 total, 7 up, 8 in 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.574235+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.574756+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.577015+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.577179+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.578710+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.579752+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.580638+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.583278+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[60360]: cluster 2023-07-29T17:15:38.583874+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.571246+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.573936+0000 mon.a (mon.0) 436 : cluster 0 osdmap e90: 8 total, 7 up, 8 in 2023-07-29T17:15:39.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.574235+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.574756+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.577015+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.577179+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.578710+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.579752+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.580638+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.583278+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:39 smithi078 bash[58614]: cluster 2023-07-29T17:15:38.583874+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.571246+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.573936+0000 mon.a (mon.0) 436 : cluster 0 osdmap e90: 8 total, 7 up, 8 in 2023-07-29T17:15:39.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.574235+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.574756+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.577015+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.577179+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.578710+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.579752+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.580638+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.583278+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:39.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:39 smithi129 bash[53756]: cluster 2023-07-29T17:15:38.583874+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e90 with expected crc 2023-07-29T17:15:40.191 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:39 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.2.service: Succeeded. 2023-07-29T17:15:40.191 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:39 smithi078 systemd[1]: Stopped Ceph osd.2 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:15:40.442 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:40 smithi078 systemd[1]: Started Ceph osd.2 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:15:40.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:40 smithi129 bash[53756]: cluster 2023-07-29T17:15:39.496220+0000 mgr.y (mgr.25169) 187 : cluster 0 pgmap v114: 161 pgs: 10 active+undersized+degraded, 3 stale+active+clean, 29 active+undersized, 119 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 74/999 objects degraded (7.407%) 2023-07-29T17:15:40.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:40 smithi129 bash[53756]: cluster 2023-07-29T17:15:39.691403+0000 mon.a (mon.0) 437 : cluster 3 Health check failed: Degraded data redundancy: 74/999 objects degraded (7.407%), 10 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:40.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:40 smithi129 bash[53756]: audit 2023-07-29T17:15:40.205354+0000 mon.a (mon.0) 438 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:40.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:40 smithi129 bash[53756]: audit 2023-07-29T17:15:40.212739+0000 mon.a (mon.0) 439 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:40.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[58614]: cluster 2023-07-29T17:15:39.496220+0000 mgr.y (mgr.25169) 187 : cluster 0 pgmap v114: 161 pgs: 10 active+undersized+degraded, 3 stale+active+clean, 29 active+undersized, 119 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 74/999 objects degraded (7.407%) 2023-07-29T17:15:40.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[58614]: cluster 2023-07-29T17:15:39.691403+0000 mon.a (mon.0) 437 : cluster 3 Health check failed: Degraded data redundancy: 74/999 objects degraded (7.407%), 10 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:40.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[58614]: audit 2023-07-29T17:15:40.205354+0000 mon.a (mon.0) 438 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:40.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[58614]: audit 2023-07-29T17:15:40.212739+0000 mon.a (mon.0) 439 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:40.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[60360]: cluster 2023-07-29T17:15:39.496220+0000 mgr.y (mgr.25169) 187 : cluster 0 pgmap v114: 161 pgs: 10 active+undersized+degraded, 3 stale+active+clean, 29 active+undersized, 119 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 74/999 objects degraded (7.407%) 2023-07-29T17:15:40.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[60360]: cluster 2023-07-29T17:15:39.691403+0000 mon.a (mon.0) 437 : cluster 3 Health check failed: Degraded data redundancy: 74/999 objects degraded (7.407%), 10 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:40.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[60360]: audit 2023-07-29T17:15:40.205354+0000 mon.a (mon.0) 438 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:40.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:40 smithi078 bash[60360]: audit 2023-07-29T17:15:40.212739+0000 mon.a (mon.0) 439 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:42.442 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-29T17:15:42.443 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-07-29T17:15:42.443 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-07-29T17:15:42.443 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-29T17:15:42.443 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2023-07-29T17:15:42.443 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-29T17:15:42.443 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[67827]: --> ceph-volume raw activate successful for osd ID: 2 2023-07-29T17:15:42.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[60360]: cluster 2023-07-29T17:15:41.497335+0000 mgr.y (mgr.25169) 188 : cluster 0 pgmap v115: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:42.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:42 smithi078 bash[58614]: cluster 2023-07-29T17:15:41.497335+0000 mgr.y (mgr.25169) 188 : cluster 0 pgmap v115: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:42.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:42 smithi129 bash[53756]: cluster 2023-07-29T17:15:41.497335+0000 mgr.y (mgr.25169) 188 : cluster 0 pgmap v115: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:44.192 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:44 smithi078 bash[68132]: debug 2023-07-29T17:15:44.002+0000 7fd6f89d4700 1 -- 172.21.15.78:0/2929652002 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555e161884e0 con 0x555e161e3000 2023-07-29T17:15:44.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:44 smithi078 bash[58614]: cluster 2023-07-29T17:15:43.498285+0000 mgr.y (mgr.25169) 189 : cluster 0 pgmap v116: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:44.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:44 smithi078 bash[60360]: cluster 2023-07-29T17:15:43.498285+0000 mgr.y (mgr.25169) 189 : cluster 0 pgmap v116: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:44.943 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:44 smithi078 bash[68132]: debug 2023-07-29T17:15:44.550+0000 7fd701b59640 -1 Falling back to public interface 2023-07-29T17:15:45.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:44 smithi129 bash[53756]: cluster 2023-07-29T17:15:43.498285+0000 mgr.y (mgr.25169) 189 : cluster 0 pgmap v116: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:47.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:46 smithi129 bash[53756]: cluster 2023-07-29T17:15:45.500214+0000 mgr.y (mgr.25169) 190 : cluster 0 pgmap v117: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:47.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:46 smithi078 bash[58614]: cluster 2023-07-29T17:15:45.500214+0000 mgr.y (mgr.25169) 190 : cluster 0 pgmap v117: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:47.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:46 smithi078 bash[60360]: cluster 2023-07-29T17:15:45.500214+0000 mgr.y (mgr.25169) 190 : cluster 0 pgmap v117: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:47.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:47 smithi078 bash[58614]: audit 2023-07-29T17:15:47.622812+0000 mon.a (mon.0) 440 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:15:47.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:47 smithi078 bash[60360]: audit 2023-07-29T17:15:47.622812+0000 mon.a (mon.0) 440 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:15:47.943 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:47 smithi078 bash[68132]: debug 2023-07-29T17:15:47.614+0000 7fd701b59640 -1 osd.2 88 log_to_monitors true 2023-07-29T17:15:47.943 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:15:47 smithi078 bash[68132]: debug 2023-07-29T17:15:47.766+0000 7fd6f17c3700 -1 osd.2 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:15:48.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:47 smithi129 bash[53756]: audit 2023-07-29T17:15:47.622812+0000 mon.a (mon.0) 440 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T17:15:49.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:47.500805+0000 mgr.y (mgr.25169) 191 : cluster 0 pgmap v118: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:49.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: audit 2023-07-29T17:15:47.712622+0000 mon.a (mon.0) 441 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T17:15:49.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:47.712828+0000 mon.a (mon.0) 442 : cluster 0 osdmap e91: 8 total, 7 up, 8 in 2023-07-29T17:15:49.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:47.713323+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: audit 2023-07-29T17:15:47.715090+0000 mon.a (mon.0) 443 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:15:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:47.815733+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: audit 2023-07-29T17:15:47.868007+0000 mon.a (mon.0) 444 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: audit 2023-07-29T17:15:47.880055+0000 mon.a (mon.0) 445 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.035189+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.037641+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: audit 2023-07-29T17:15:48.674167+0000 mon.a (mon.0) 446 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: audit 2023-07-29T17:15:48.685447+0000 mon.a (mon.0) 447 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:48 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.691344+0000 mon.a (mon.0) 448 : cluster 3 Health check update: Degraded data redundancy: 85/999 objects degraded (8.509%), 11 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:49.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:47.500805+0000 mgr.y (mgr.25169) 191 : cluster 0 pgmap v118: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: audit 2023-07-29T17:15:47.712622+0000 mon.a (mon.0) 441 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T17:15:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:47.712828+0000 mon.a (mon.0) 442 : cluster 0 osdmap e91: 8 total, 7 up, 8 in 2023-07-29T17:15:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:47.713323+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: audit 2023-07-29T17:15:47.715090+0000 mon.a (mon.0) 443 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:15:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:47.815733+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: audit 2023-07-29T17:15:47.868007+0000 mon.a (mon.0) 444 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: audit 2023-07-29T17:15:47.880055+0000 mon.a (mon.0) 445 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.035189+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.037641+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: audit 2023-07-29T17:15:48.674167+0000 mon.a (mon.0) 446 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: audit 2023-07-29T17:15:48.685447+0000 mon.a (mon.0) 447 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.691344+0000 mon.a (mon.0) 448 : cluster 3 Health check update: Degraded data redundancy: 85/999 objects degraded (8.509%), 11 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:47.500805+0000 mgr.y (mgr.25169) 191 : cluster 0 pgmap v118: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-29T17:15:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: audit 2023-07-29T17:15:47.712622+0000 mon.a (mon.0) 441 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T17:15:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:47.712828+0000 mon.a (mon.0) 442 : cluster 0 osdmap e91: 8 total, 7 up, 8 in 2023-07-29T17:15:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:47.713323+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: audit 2023-07-29T17:15:47.715090+0000 mon.a (mon.0) 443 : audit 1 from='osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:15:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:47.815733+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: audit 2023-07-29T17:15:47.868007+0000 mon.a (mon.0) 444 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: audit 2023-07-29T17:15:47.880055+0000 mon.a (mon.0) 445 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.035189+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.037641+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: audit 2023-07-29T17:15:48.674167+0000 mon.a (mon.0) 446 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: audit 2023-07-29T17:15:48.685447+0000 mon.a (mon.0) 447 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:15:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:48 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.691344+0000 mon.a (mon.0) 448 : cluster 3 Health check update: Degraded data redundancy: 85/999 objects degraded (8.509%), 11 pgs degraded (PG_DEGRADED) 2023-07-29T17:15:50.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.025810+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.709445+0000 mon.a (mon.0) 449 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.712874+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e92 with expected crc 2023-07-29T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.715842+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-29T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.718277+0000 mon.a (mon.0) 450 : cluster 1 osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584] boot 2023-07-29T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: cluster 2023-07-29T17:15:48.718402+0000 mon.a (mon.0) 451 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2023-07-29T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:49 smithi129 bash[53756]: audit 2023-07-29T17:15:48.719117+0000 mon.a (mon.0) 452 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:15:50.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.025810+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:50.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.709445+0000 mon.a (mon.0) 449 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:15:50.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.712874+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e92 with expected crc 2023-07-29T17:15:50.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.715842+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-29T17:15:50.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.718277+0000 mon.a (mon.0) 450 : cluster 1 osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584] boot 2023-07-29T17:15:50.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: cluster 2023-07-29T17:15:48.718402+0000 mon.a (mon.0) 451 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2023-07-29T17:15:50.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[60360]: audit 2023-07-29T17:15:48.719117+0000 mon.a (mon.0) 452 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:15:50.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.025810+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-29T17:15:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.709445+0000 mon.a (mon.0) 449 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:15:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.712874+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e92 with expected crc 2023-07-29T17:15:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.715842+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-29T17:15:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.718277+0000 mon.a (mon.0) 450 : cluster 1 osd.2 [v2:172.21.15.78:6818/2817626584,v1:172.21.15.78:6819/2817626584] boot 2023-07-29T17:15:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: cluster 2023-07-29T17:15:48.718402+0000 mon.a (mon.0) 451 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2023-07-29T17:15:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:49 smithi078 bash[58614]: audit 2023-07-29T17:15:48.719117+0000 mon.a (mon.0) 452 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T17:15:50.232 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:15:50.972 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:15:50.973 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5m) 3s ago 11m 10.5M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:15:50.973 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (5m) 2m ago 11m 26.1M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:15:50.973 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 2m ago 9m - - 2023-07-29T17:15:50.973 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (3m) 2m ago 16m 428M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:15:50.973 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (4m) 3s ago 18m 472M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:15:50.974 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3m) 3s ago 18m 59.8M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:15:50.974 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (2m) 2m ago 16m 37.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:15:50.974 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2m) 3s ago 16m 49.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:15:50.974 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5m) 3s ago 12m 8775k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:15:50.974 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (5m) 2m ago 12m 3644k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (66s) 3s ago 15m 49.1M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (37s) 3s ago 15m 49.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8s) 3s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (14m) 3s ago 14m 58.0M 4096M 16.2.0 24ecd6d5f14c 2a0b31fcb688 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (14m) 2m ago 14m 55.5M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (13m) 2m ago 13m 53.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (13m) 2m ago 13m 53.3M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:15:50.975 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (12m) 2m ago 13m 56.5M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:15:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 2m ago 12m - - 2023-07-29T17:15:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (10m) 3s ago 10m 55.4M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:15:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (9m) 2m ago 9m 54.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:15:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (9m) 3s ago 9m 55.6M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:15:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (9m) 2m ago 9m 54.5M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:15:51.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.502325+0000 mgr.y (mgr.25169) 192 : cluster 0 pgmap v121: 161 pgs: 29 peering, 1 active+undersized+degraded, 17 active+undersized, 114 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 11/999 objects degraded (1.101%) 2023-07-29T17:15:51.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.731287+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.735540+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.736881+0000 mon.a (mon.0) 453 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2023-07-29T17:15:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.737188+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.739868+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.739893+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.740688+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:50 smithi129 bash[53756]: cluster 2023-07-29T17:15:49.741178+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.502325+0000 mgr.y (mgr.25169) 192 : cluster 0 pgmap v121: 161 pgs: 29 peering, 1 active+undersized+degraded, 17 active+undersized, 114 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 11/999 objects degraded (1.101%) 2023-07-29T17:15:51.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.731287+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.735540+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.736881+0000 mon.a (mon.0) 453 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2023-07-29T17:15:51.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.737188+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.739868+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.739893+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.740688+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[60360]: cluster 2023-07-29T17:15:49.741178+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.502325+0000 mgr.y (mgr.25169) 192 : cluster 0 pgmap v121: 161 pgs: 29 peering, 1 active+undersized+degraded, 17 active+undersized, 114 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 11/999 objects degraded (1.101%) 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.731287+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.735540+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.736881+0000 mon.a (mon.0) 453 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2023-07-29T17:15:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.737188+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.739868+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.739893+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.740688+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:50 smithi078 bash[58614]: cluster 2023-07-29T17:15:49.741178+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-29T17:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2023-07-29T17:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 9, 2023-07-29T17:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "8/23 daemons upgraded", 2023-07-29T17:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:15:52.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:51 smithi129 bash[53756]: audit 2023-07-29T17:15:50.196079+0000 mgr.y (mgr.25169) 193 : audit 0 from='client.44264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:51 smithi129 bash[53756]: audit 2023-07-29T17:15:50.569850+0000 mgr.y (mgr.25169) 194 : audit 0 from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:51 smithi129 bash[53756]: audit 2023-07-29T17:15:50.929315+0000 mgr.y (mgr.25169) 195 : audit 0 from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:51 smithi129 bash[53756]: audit 2023-07-29T17:15:51.386270+0000 mon.a (mon.0) 454 : audit 0 from='client.34452 172.21.15.78:0/752334309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:52.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[58614]: audit 2023-07-29T17:15:50.196079+0000 mgr.y (mgr.25169) 193 : audit 0 from='client.44264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[58614]: audit 2023-07-29T17:15:50.569850+0000 mgr.y (mgr.25169) 194 : audit 0 from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[58614]: audit 2023-07-29T17:15:50.929315+0000 mgr.y (mgr.25169) 195 : audit 0 from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[58614]: audit 2023-07-29T17:15:51.386270+0000 mon.a (mon.0) 454 : audit 0 from='client.34452 172.21.15.78:0/752334309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:52.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[60360]: audit 2023-07-29T17:15:50.196079+0000 mgr.y (mgr.25169) 193 : audit 0 from='client.44264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[60360]: audit 2023-07-29T17:15:50.569850+0000 mgr.y (mgr.25169) 194 : audit 0 from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[60360]: audit 2023-07-29T17:15:50.929315+0000 mgr.y (mgr.25169) 195 : audit 0 from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:52.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:51 smithi078 bash[60360]: audit 2023-07-29T17:15:51.386270+0000 mon.a (mon.0) 454 : audit 0 from='client.34452 172.21.15.78:0/752334309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:15:52.200 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:15:52.200 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:15:52.200 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:15:52.200 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:15:53.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:52 smithi129 bash[53756]: cluster 2023-07-29T17:15:51.503235+0000 mgr.y (mgr.25169) 196 : cluster 0 pgmap v123: 161 pgs: 34 peering, 13 active+undersized, 114 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:53.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:52 smithi129 bash[53756]: cluster 2023-07-29T17:15:51.732206+0000 mon.a (mon.0) 455 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded) 2023-07-29T17:15:53.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:52 smithi129 bash[53756]: audit 2023-07-29T17:15:51.743283+0000 mgr.y (mgr.25169) 197 : audit 0 from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:53.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:52 smithi129 bash[53756]: audit 2023-07-29T17:15:52.178894+0000 mon.c (mon.2) 9 : audit 0 from='client.? 172.21.15.78:0/298919026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:15:53.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[58614]: cluster 2023-07-29T17:15:51.503235+0000 mgr.y (mgr.25169) 196 : cluster 0 pgmap v123: 161 pgs: 34 peering, 13 active+undersized, 114 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[58614]: cluster 2023-07-29T17:15:51.732206+0000 mon.a (mon.0) 455 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded) 2023-07-29T17:15:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[58614]: audit 2023-07-29T17:15:51.743283+0000 mgr.y (mgr.25169) 197 : audit 0 from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[58614]: audit 2023-07-29T17:15:52.178894+0000 mon.c (mon.2) 9 : audit 0 from='client.? 172.21.15.78:0/298919026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:15:53.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[60360]: cluster 2023-07-29T17:15:51.503235+0000 mgr.y (mgr.25169) 196 : cluster 0 pgmap v123: 161 pgs: 34 peering, 13 active+undersized, 114 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:53.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[60360]: cluster 2023-07-29T17:15:51.732206+0000 mon.a (mon.0) 455 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded) 2023-07-29T17:15:53.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[60360]: audit 2023-07-29T17:15:51.743283+0000 mgr.y (mgr.25169) 197 : audit 0 from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:15:53.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:52 smithi078 bash[60360]: audit 2023-07-29T17:15:52.178894+0000 mon.c (mon.2) 9 : audit 0 from='client.? 172.21.15.78:0/298919026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:15:55.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:54 smithi078 bash[60360]: cluster 2023-07-29T17:15:53.504160+0000 mgr.y (mgr.25169) 198 : cluster 0 pgmap v124: 161 pgs: 34 peering, 127 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:55.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:54 smithi078 bash[58614]: cluster 2023-07-29T17:15:53.504160+0000 mgr.y (mgr.25169) 198 : cluster 0 pgmap v124: 161 pgs: 34 peering, 127 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:55.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:54 smithi129 bash[53756]: cluster 2023-07-29T17:15:53.504160+0000 mgr.y (mgr.25169) 198 : cluster 0 pgmap v124: 161 pgs: 34 peering, 127 active+clean; 604 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:15:57.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:56 smithi129 bash[53756]: cluster 2023-07-29T17:15:55.505829+0000 mgr.y (mgr.25169) 199 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2023-07-29T17:15:57.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:56 smithi078 bash[58614]: cluster 2023-07-29T17:15:55.505829+0000 mgr.y (mgr.25169) 199 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2023-07-29T17:15:57.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:56 smithi078 bash[60360]: cluster 2023-07-29T17:15:55.505829+0000 mgr.y (mgr.25169) 199 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2023-07-29T17:15:59.173 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:15:58 smithi129 bash[53756]: cluster 2023-07-29T17:15:57.506651+0000 mgr.y (mgr.25169) 200 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2023-07-29T17:15:59.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:15:58 smithi078 bash[60360]: cluster 2023-07-29T17:15:57.506651+0000 mgr.y (mgr.25169) 200 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2023-07-29T17:15:59.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:15:58 smithi078 bash[58614]: cluster 2023-07-29T17:15:57.506651+0000 mgr.y (mgr.25169) 200 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2023-07-29T17:16:00.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.136073+0000 mon.a (mon.0) 456 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.147385+0000 mon.a (mon.0) 457 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.149389+0000 mon.a (mon.0) 458 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:00.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.150704+0000 mon.a (mon.0) 459 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: cephadm 2023-07-29T17:15:59.152047+0000 mgr.y (mgr.25169) 201 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: cluster 2023-07-29T17:15:59.153628+0000 mgr.y (mgr.25169) 202 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: cluster 2023-07-29T17:15:59.154260+0000 mgr.y (mgr.25169) 203 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.161329+0000 mon.a (mon.0) 460 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.247115+0000 mon.a (mon.0) 461 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.252147+0000 mon.a (mon.0) 462 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.254725+0000 mon.a (mon.0) 463 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.256595+0000 mon.a (mon.0) 464 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:00 smithi129 bash[53756]: audit 2023-07-29T17:15:59.258760+0000 mon.a (mon.0) 465 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-29T17:16:00.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.136073+0000 mon.a (mon.0) 456 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.147385+0000 mon.a (mon.0) 457 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.149389+0000 mon.a (mon.0) 458 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:00.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.150704+0000 mon.a (mon.0) 459 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:16:00.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: cephadm 2023-07-29T17:15:59.152047+0000 mgr.y (mgr.25169) 201 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:16:00.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: cluster 2023-07-29T17:15:59.153628+0000 mgr.y (mgr.25169) 202 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: cluster 2023-07-29T17:15:59.154260+0000 mgr.y (mgr.25169) 203 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.161329+0000 mon.a (mon.0) 460 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.247115+0000 mon.a (mon.0) 461 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.252147+0000 mon.a (mon.0) 462 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.254725+0000 mon.a (mon.0) 463 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.256595+0000 mon.a (mon.0) 464 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[58614]: audit 2023-07-29T17:15:59.258760+0000 mon.a (mon.0) 465 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.136073+0000 mon.a (mon.0) 456 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.147385+0000 mon.a (mon.0) 457 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.149389+0000 mon.a (mon.0) 458 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.150704+0000 mon.a (mon.0) 459 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: cephadm 2023-07-29T17:15:59.152047+0000 mgr.y (mgr.25169) 201 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: cluster 2023-07-29T17:15:59.153628+0000 mgr.y (mgr.25169) 202 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2023-07-29T17:16:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: cluster 2023-07-29T17:15:59.154260+0000 mgr.y (mgr.25169) 203 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2023-07-29T17:16:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.161329+0000 mon.a (mon.0) 460 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.247115+0000 mon.a (mon.0) 461 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:16:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.252147+0000 mon.a (mon.0) 462 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.254725+0000 mon.a (mon.0) 463 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.256595+0000 mon.a (mon.0) 464 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:00 smithi078 bash[60360]: audit 2023-07-29T17:15:59.258760+0000 mon.a (mon.0) 465 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-29T17:16:01.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:01 smithi129 bash[53756]: audit 2023-07-29T17:15:59.259082+0000 mgr.y (mgr.25169) 204 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-29T17:16:01.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:01 smithi129 bash[53756]: cephadm 2023-07-29T17:15:59.261225+0000 mgr.y (mgr.25169) 205 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-07-29T17:16:01.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:01 smithi078 bash[58614]: audit 2023-07-29T17:15:59.259082+0000 mgr.y (mgr.25169) 204 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-29T17:16:01.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:01 smithi078 bash[58614]: cephadm 2023-07-29T17:15:59.261225+0000 mgr.y (mgr.25169) 205 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-07-29T17:16:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:01 smithi078 bash[60360]: audit 2023-07-29T17:15:59.259082+0000 mgr.y (mgr.25169) 204 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-29T17:16:01.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:01 smithi078 bash[60360]: cephadm 2023-07-29T17:15:59.261225+0000 mgr.y (mgr.25169) 205 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-07-29T17:16:02.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[58614]: cluster 2023-07-29T17:16:01.155870+0000 mgr.y (mgr.25169) 206 : cluster 0 pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2023-07-29T17:16:02.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[58614]: audit 2023-07-29T17:16:01.541592+0000 mon.a (mon.0) 466 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:02.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[58614]: audit 2023-07-29T17:16:01.543106+0000 mon.a (mon.0) 467 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:16:02.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[58614]: audit 2023-07-29T17:16:01.544211+0000 mon.a (mon.0) 468 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:02.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[60360]: cluster 2023-07-29T17:16:01.155870+0000 mgr.y (mgr.25169) 206 : cluster 0 pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2023-07-29T17:16:02.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[60360]: audit 2023-07-29T17:16:01.541592+0000 mon.a (mon.0) 466 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:02.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[60360]: audit 2023-07-29T17:16:01.543106+0000 mon.a (mon.0) 467 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:16:02.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:02 smithi078 bash[60360]: audit 2023-07-29T17:16:01.544211+0000 mon.a (mon.0) 468 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:02.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:02 smithi129 bash[53756]: cluster 2023-07-29T17:16:01.155870+0000 mgr.y (mgr.25169) 206 : cluster 0 pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2023-07-29T17:16:02.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:02 smithi129 bash[53756]: audit 2023-07-29T17:16:01.541592+0000 mon.a (mon.0) 466 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:02.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:02 smithi129 bash[53756]: audit 2023-07-29T17:16:01.543106+0000 mon.a (mon.0) 467 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T17:16:02.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:02 smithi129 bash[53756]: audit 2023-07-29T17:16:01.544211+0000 mon.a (mon.0) 468 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:03.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:03 smithi078 bash[58614]: cephadm 2023-07-29T17:16:01.531243+0000 mgr.y (mgr.25169) 207 : cephadm 1 Upgrade: Updating osd.3 2023-07-29T17:16:03.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:03 smithi078 bash[58614]: cephadm 2023-07-29T17:16:01.549740+0000 mgr.y (mgr.25169) 208 : cephadm 1 Deploying daemon osd.3 on smithi078 2023-07-29T17:16:03.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:03 smithi078 bash[60360]: cephadm 2023-07-29T17:16:01.531243+0000 mgr.y (mgr.25169) 207 : cephadm 1 Upgrade: Updating osd.3 2023-07-29T17:16:03.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:03 smithi078 bash[60360]: cephadm 2023-07-29T17:16:01.549740+0000 mgr.y (mgr.25169) 208 : cephadm 1 Deploying daemon osd.3 on smithi078 2023-07-29T17:16:03.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:03 smithi129 bash[53756]: cephadm 2023-07-29T17:16:01.531243+0000 mgr.y (mgr.25169) 207 : cephadm 1 Upgrade: Updating osd.3 2023-07-29T17:16:03.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:03 smithi129 bash[53756]: cephadm 2023-07-29T17:16:01.549740+0000 mgr.y (mgr.25169) 208 : cephadm 1 Deploying daemon osd.3 on smithi078 2023-07-29T17:16:04.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:04 smithi078 bash[60360]: cluster 2023-07-29T17:16:03.156683+0000 mgr.y (mgr.25169) 209 : cluster 0 pgmap v130: 161 pgs: 161 active+clean; 604 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2023-07-29T17:16:04.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:04 smithi078 bash[60360]: audit 2023-07-29T17:16:03.361042+0000 mon.a (mon.0) 469 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:04.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:04 smithi078 bash[58614]: cluster 2023-07-29T17:16:03.156683+0000 mgr.y (mgr.25169) 209 : cluster 0 pgmap v130: 161 pgs: 161 active+clean; 604 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2023-07-29T17:16:04.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:04 smithi078 bash[58614]: audit 2023-07-29T17:16:03.361042+0000 mon.a (mon.0) 469 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:04.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:04 smithi129 bash[53756]: cluster 2023-07-29T17:16:03.156683+0000 mgr.y (mgr.25169) 209 : cluster 0 pgmap v130: 161 pgs: 161 active+clean; 604 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2023-07-29T17:16:04.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:04 smithi129 bash[53756]: audit 2023-07-29T17:16:03.361042+0000 mon.a (mon.0) 469 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:06.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:06 smithi078 bash[60360]: cluster 2023-07-29T17:16:05.158420+0000 mgr.y (mgr.25169) 210 : cluster 0 pgmap v131: 161 pgs: 161 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2023-07-29T17:16:06.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:06 smithi078 bash[58614]: cluster 2023-07-29T17:16:05.158420+0000 mgr.y (mgr.25169) 210 : cluster 0 pgmap v131: 161 pgs: 161 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2023-07-29T17:16:06.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:06 smithi129 bash[53756]: cluster 2023-07-29T17:16:05.158420+0000 mgr.y (mgr.25169) 210 : cluster 0 pgmap v131: 161 pgs: 161 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2023-07-29T17:16:06.854 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:06 smithi078 systemd[1]: Stopping Ceph osd.3 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:16:06.854 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:06 smithi078 bash[39179]: debug 2023-07-29T17:16:06.598+0000 7fa39fd7f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:16:06.854 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:06 smithi078 bash[39179]: debug 2023-07-29T17:16:06.598+0000 7fa39fd7f700 -1 osd.3 93 *** Got signal Terminated *** 2023-07-29T17:16:06.855 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:06 smithi078 bash[39179]: debug 2023-07-29T17:16:06.598+0000 7fa39fd7f700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:16:07.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.864440+0000 mon.a (mon.0) 470 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-29T17:16:07.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.864509+0000 mon.a (mon.0) 471 : cluster 1 osd.3 failed (root=default,host=smithi078) (connection refused reported by osd.1) 2023-07-29T17:16:07.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.864848+0000 mon.a (mon.0) 472 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-29T17:16:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.865141+0000 mon.a (mon.0) 473 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-29T17:16:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.865637+0000 mon.a (mon.0) 474 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-29T17:16:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.866119+0000 mon.a (mon.0) 475 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-29T17:16:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.866543+0000 mon.a (mon.0) 476 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-29T17:16:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.866802+0000 mon.a (mon.0) 477 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-29T17:16:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.867438+0000 mon.a (mon.0) 478 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.867901+0000 mon.a (mon.0) 479 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.868134+0000 mon.a (mon.0) 480 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.868797+0000 mon.a (mon.0) 481 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.869275+0000 mon.a (mon.0) 482 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.869794+0000 mon.a (mon.0) 483 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[58614]: cluster 2023-07-29T17:16:06.870317+0000 mon.a (mon.0) 484 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-29T17:16:07.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.864440+0000 mon.a (mon.0) 470 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.864509+0000 mon.a (mon.0) 471 : cluster 1 osd.3 failed (root=default,host=smithi078) (connection refused reported by osd.1) 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.864848+0000 mon.a (mon.0) 472 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.865141+0000 mon.a (mon.0) 473 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.865637+0000 mon.a (mon.0) 474 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.866119+0000 mon.a (mon.0) 475 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.866543+0000 mon.a (mon.0) 476 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-29T17:16:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.866802+0000 mon.a (mon.0) 477 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.867438+0000 mon.a (mon.0) 478 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.867901+0000 mon.a (mon.0) 479 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.868134+0000 mon.a (mon.0) 480 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.868797+0000 mon.a (mon.0) 481 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.869275+0000 mon.a (mon.0) 482 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.869794+0000 mon.a (mon.0) 483 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-29T17:16:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[60360]: cluster 2023-07-29T17:16:06.870317+0000 mon.a (mon.0) 484 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-29T17:16:07.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.864440+0000 mon.a (mon.0) 470 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-29T17:16:07.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.864509+0000 mon.a (mon.0) 471 : cluster 1 osd.3 failed (root=default,host=smithi078) (connection refused reported by osd.1) 2023-07-29T17:16:07.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.864848+0000 mon.a (mon.0) 472 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-29T17:16:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.865141+0000 mon.a (mon.0) 473 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-29T17:16:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.865637+0000 mon.a (mon.0) 474 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-29T17:16:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.866119+0000 mon.a (mon.0) 475 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-29T17:16:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.866543+0000 mon.a (mon.0) 476 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-29T17:16:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.866802+0000 mon.a (mon.0) 477 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-29T17:16:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.867438+0000 mon.a (mon.0) 478 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-29T17:16:07.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.867901+0000 mon.a (mon.0) 479 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-29T17:16:07.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.868134+0000 mon.a (mon.0) 480 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-29T17:16:07.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.868797+0000 mon.a (mon.0) 481 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-29T17:16:07.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.869275+0000 mon.a (mon.0) 482 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-29T17:16:07.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.869794+0000 mon.a (mon.0) 483 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-29T17:16:07.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:07 smithi129 bash[53756]: cluster 2023-07-29T17:16:06.870317+0000 mon.a (mon.0) 484 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-29T17:16:07.942 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:07 smithi078 bash[69862]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.3 2023-07-29T17:16:08.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.159268+0000 mgr.y (mgr.25169) 211 : cluster 0 pgmap v132: 161 pgs: 161 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2023-07-29T17:16:08.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.167143+0000 mon.a (mon.0) 485 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:16:08.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.184703+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.186158+0000 mon.a (mon.0) 486 : cluster 0 osdmap e94: 8 total, 7 up, 8 in 2023-07-29T17:16:08.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.188585+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.188820+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.190803+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.191252+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.191871+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.193212+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.193236+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:08 smithi129 bash[53756]: cluster 2023-07-29T17:16:07.194332+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.159268+0000 mgr.y (mgr.25169) 211 : cluster 0 pgmap v132: 161 pgs: 161 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.167143+0000 mon.a (mon.0) 485 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.184703+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.186158+0000 mon.a (mon.0) 486 : cluster 0 osdmap e94: 8 total, 7 up, 8 in 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.188585+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.188820+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.190803+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.191252+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.191871+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.193212+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.193236+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[58614]: cluster 2023-07-29T17:16:07.194332+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.159268+0000 mgr.y (mgr.25169) 211 : cluster 0 pgmap v132: 161 pgs: 161 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2023-07-29T17:16:08.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.167143+0000 mon.a (mon.0) 485 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.184703+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.186158+0000 mon.a (mon.0) 486 : cluster 0 osdmap e94: 8 total, 7 up, 8 in 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.188585+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.188820+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.190803+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.191252+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.191871+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.193212+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.193236+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:08.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:08 smithi078 bash[60360]: cluster 2023-07-29T17:16:07.194332+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e94 with expected crc 2023-07-29T17:16:09.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.185665+0000 mon.a (mon.0) 487 : cluster 0 osdmap e95: 8 total, 7 up, 8 in 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.186155+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.186532+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.188865+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.189177+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.189713+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.190830+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.191586+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.193637+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.195236+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.195494+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.197908+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.198949+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.199801+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.200512+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[60360]: cluster 2023-07-29T17:16:08.201741+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.185665+0000 mon.a (mon.0) 487 : cluster 0 osdmap e95: 8 total, 7 up, 8 in 2023-07-29T17:16:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.186155+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.186532+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.188865+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.189177+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.189713+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.190830+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.191586+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.193637+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.195236+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.195494+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.197908+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.198949+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.199801+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.200512+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:09 smithi078 bash[58614]: cluster 2023-07-29T17:16:08.201741+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.185665+0000 mon.a (mon.0) 487 : cluster 0 osdmap e95: 8 total, 7 up, 8 in 2023-07-29T17:16:09.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.186155+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.186532+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.188865+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.189177+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.189713+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.190830+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.191586+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.193637+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.195236+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.195494+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.197908+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.198949+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.199801+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.200512+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:09.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:09 smithi129 bash[53756]: cluster 2023-07-29T17:16:08.201741+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e95 with expected crc 2023-07-29T17:16:10.367 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:10 smithi078 bash[58614]: cluster 2023-07-29T17:16:09.160062+0000 mgr.y (mgr.25169) 212 : cluster 0 pgmap v135: 161 pgs: 6 peering, 26 stale+active+clean, 129 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2023-07-29T17:16:10.367 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:10 smithi078 bash[60360]: cluster 2023-07-29T17:16:09.160062+0000 mgr.y (mgr.25169) 212 : cluster 0 pgmap v135: 161 pgs: 6 peering, 26 stale+active+clean, 129 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2023-07-29T17:16:10.368 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:10 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.3.service: Succeeded. 2023-07-29T17:16:10.368 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:10 smithi078 systemd[1]: Stopped Ceph osd.3 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:16:10.368 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:10 smithi078 systemd[1]: Started Ceph osd.3 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:16:10.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:10 smithi129 bash[53756]: cluster 2023-07-29T17:16:09.160062+0000 mgr.y (mgr.25169) 212 : cluster 0 pgmap v135: 161 pgs: 6 peering, 26 stale+active+clean, 129 active+clean; 604 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2023-07-29T17:16:11.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:11 smithi129 bash[53756]: audit 2023-07-29T17:16:10.382804+0000 mon.a (mon.0) 488 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:11.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:11 smithi129 bash[53756]: audit 2023-07-29T17:16:10.390123+0000 mon.a (mon.0) 489 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:11.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:11 smithi129 bash[53756]: cluster 2023-07-29T17:16:11.162050+0000 mgr.y (mgr.25169) 213 : cluster 0 pgmap v136: 161 pgs: 24 active+undersized+degraded, 6 peering, 36 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 135 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:11.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:11 smithi078 bash[60360]: audit 2023-07-29T17:16:10.382804+0000 mon.a (mon.0) 488 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:11.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:11 smithi078 bash[60360]: audit 2023-07-29T17:16:10.390123+0000 mon.a (mon.0) 489 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:11.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:11 smithi078 bash[60360]: cluster 2023-07-29T17:16:11.162050+0000 mgr.y (mgr.25169) 213 : cluster 0 pgmap v136: 161 pgs: 24 active+undersized+degraded, 6 peering, 36 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 135 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:11.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:11 smithi078 bash[58614]: audit 2023-07-29T17:16:10.382804+0000 mon.a (mon.0) 488 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:11.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:11 smithi078 bash[58614]: audit 2023-07-29T17:16:10.390123+0000 mon.a (mon.0) 489 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:11.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:11 smithi078 bash[58614]: cluster 2023-07-29T17:16:11.162050+0000 mgr.y (mgr.25169) 213 : cluster 0 pgmap v136: 161 pgs: 24 active+undersized+degraded, 6 peering, 36 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 135 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:12.407 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-29T17:16:12.407 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-07-29T17:16:12.407 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-07-29T17:16:12.408 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-29T17:16:12.408 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2023-07-29T17:16:12.408 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-29T17:16:12.408 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[70129]: --> ceph-volume raw activate successful for osd ID: 3 2023-07-29T17:16:12.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:12 smithi129 bash[53756]: cluster 2023-07-29T17:16:11.389549+0000 mon.a (mon.0) 490 : cluster 3 Health check failed: Degraded data redundancy: 138/999 objects degraded (13.814%), 24 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:12.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[58614]: cluster 2023-07-29T17:16:11.389549+0000 mon.a (mon.0) 490 : cluster 3 Health check failed: Degraded data redundancy: 138/999 objects degraded (13.814%), 24 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:12.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:12 smithi078 bash[60360]: cluster 2023-07-29T17:16:11.389549+0000 mon.a (mon.0) 490 : cluster 3 Health check failed: Degraded data redundancy: 138/999 objects degraded (13.814%), 24 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:13.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:13 smithi129 bash[53756]: cluster 2023-07-29T17:16:13.163292+0000 mgr.y (mgr.25169) 214 : cluster 0 pgmap v137: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 85 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:13.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:13 smithi078 bash[58614]: cluster 2023-07-29T17:16:13.163292+0000 mgr.y (mgr.25169) 214 : cluster 0 pgmap v137: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 85 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:13.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:13 smithi078 bash[60360]: cluster 2023-07-29T17:16:13.163292+0000 mgr.y (mgr.25169) 214 : cluster 0 pgmap v137: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 85 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:14.193 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:13 smithi078 bash[70437]: debug 2023-07-29T17:16:13.930+0000 7fc8a4ec7700 1 -- 172.21.15.78:0/2381413278 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605320224e0 con 0x56053207d000 2023-07-29T17:16:14.692 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:14 smithi078 bash[70437]: debug 2023-07-29T17:16:14.234+0000 7fc8ae04c640 -1 Falling back to public interface 2023-07-29T17:16:15.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:15 smithi129 bash[53756]: cluster 2023-07-29T17:16:15.165273+0000 mgr.y (mgr.25169) 215 : cluster 0 pgmap v138: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 85 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:15.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:15 smithi078 bash[60360]: cluster 2023-07-29T17:16:15.165273+0000 mgr.y (mgr.25169) 215 : cluster 0 pgmap v138: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 85 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:15.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:15 smithi078 bash[58614]: cluster 2023-07-29T17:16:15.165273+0000 mgr.y (mgr.25169) 215 : cluster 0 pgmap v138: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 85 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:17.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:17 smithi129 bash[53756]: cluster 2023-07-29T17:16:17.166228+0000 mgr.y (mgr.25169) 216 : cluster 0 pgmap v139: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 68 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:17 smithi078 bash[58614]: cluster 2023-07-29T17:16:17.166228+0000 mgr.y (mgr.25169) 216 : cluster 0 pgmap v139: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 68 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:17.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:17 smithi078 bash[60360]: cluster 2023-07-29T17:16:17.166228+0000 mgr.y (mgr.25169) 216 : cluster 0 pgmap v139: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 68 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:17.694 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:17 smithi078 bash[70437]: debug 2023-07-29T17:16:17.342+0000 7fc8ae04c640 -1 osd.3 93 log_to_monitors true 2023-07-29T17:16:18.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:18 smithi129 bash[53756]: audit 2023-07-29T17:16:17.352281+0000 mon.a (mon.0) 491 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:16:18.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:18 smithi129 bash[53756]: audit 2023-07-29T17:16:17.785849+0000 mon.a (mon.0) 492 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:18.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:18 smithi129 bash[53756]: audit 2023-07-29T17:16:17.797244+0000 mon.a (mon.0) 493 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:18.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[60360]: audit 2023-07-29T17:16:17.352281+0000 mon.a (mon.0) 491 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:16:18.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[60360]: audit 2023-07-29T17:16:17.785849+0000 mon.a (mon.0) 492 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:18.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[60360]: audit 2023-07-29T17:16:17.797244+0000 mon.a (mon.0) 493 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:18.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[58614]: audit 2023-07-29T17:16:17.352281+0000 mon.a (mon.0) 491 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T17:16:18.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[58614]: audit 2023-07-29T17:16:17.785849+0000 mon.a (mon.0) 492 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:18.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[58614]: audit 2023-07-29T17:16:17.797244+0000 mon.a (mon.0) 493 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:18.694 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:16:18 smithi078 bash[70437]: debug 2023-07-29T17:16:18.286+0000 7fc89d4b5700 -1 osd.3 93 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:16:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: audit 2023-07-29T17:16:18.252019+0000 mon.a (mon.0) 494 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T17:16:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: cluster 2023-07-29T17:16:18.252069+0000 mon.a (mon.0) 495 : cluster 0 osdmap e96: 8 total, 7 up, 8 in 2023-07-29T17:16:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: cluster 2023-07-29T17:16:18.253157+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-29T17:16:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: audit 2023-07-29T17:16:18.253310+0000 mon.a (mon.0) 496 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:16:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: audit 2023-07-29T17:16:18.388517+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: cluster 2023-07-29T17:16:18.402976+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e96 with expected crc 2023-07-29T17:16:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: audit 2023-07-29T17:16:18.596042+0000 mon.a (mon.0) 498 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: audit 2023-07-29T17:16:18.606801+0000 mon.a (mon.0) 499 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:19 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.167307+0000 mgr.y (mgr.25169) 217 : cluster 0 pgmap v141: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 68 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:19.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: audit 2023-07-29T17:16:18.252019+0000 mon.a (mon.0) 494 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T17:16:19.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: cluster 2023-07-29T17:16:18.252069+0000 mon.a (mon.0) 495 : cluster 0 osdmap e96: 8 total, 7 up, 8 in 2023-07-29T17:16:19.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: cluster 2023-07-29T17:16:18.253157+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-29T17:16:19.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: audit 2023-07-29T17:16:18.253310+0000 mon.a (mon.0) 496 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:16:19.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: audit 2023-07-29T17:16:18.388517+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: cluster 2023-07-29T17:16:18.402976+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e96 with expected crc 2023-07-29T17:16:19.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: audit 2023-07-29T17:16:18.596042+0000 mon.a (mon.0) 498 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: audit 2023-07-29T17:16:18.606801+0000 mon.a (mon.0) 499 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.167307+0000 mgr.y (mgr.25169) 217 : cluster 0 pgmap v141: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 68 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:19.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: audit 2023-07-29T17:16:18.252019+0000 mon.a (mon.0) 494 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T17:16:19.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: cluster 2023-07-29T17:16:18.252069+0000 mon.a (mon.0) 495 : cluster 0 osdmap e96: 8 total, 7 up, 8 in 2023-07-29T17:16:19.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: cluster 2023-07-29T17:16:18.253157+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-29T17:16:19.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: audit 2023-07-29T17:16:18.253310+0000 mon.a (mon.0) 496 : audit 1 from='osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-29T17:16:19.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: audit 2023-07-29T17:16:18.388517+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: cluster 2023-07-29T17:16:18.402976+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e96 with expected crc 2023-07-29T17:16:19.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: audit 2023-07-29T17:16:18.596042+0000 mon.a (mon.0) 498 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: audit 2023-07-29T17:16:18.606801+0000 mon.a (mon.0) 499 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:19.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:19 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.167307+0000 mgr.y (mgr.25169) 217 : cluster 0 pgmap v141: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 68 op/s; 138/999 objects degraded (13.814%) 2023-07-29T17:16:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.249766+0000 mon.a (mon.0) 500 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:16:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.256090+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.256823+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.258227+0000 mon.a (mon.0) 501 : cluster 1 osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190] boot 2023-07-29T17:16:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.258339+0000 mon.a (mon.0) 502 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2023-07-29T17:16:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: audit 2023-07-29T17:16:19.259309+0000 mon.a (mon.0) 503 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:16:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.262057+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:19.263540+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:20 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.260254+0000 mon.a (mon.0) 504 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2023-07-29T17:16:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.249766+0000 mon.a (mon.0) 500 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:16:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.256090+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.256823+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.258227+0000 mon.a (mon.0) 501 : cluster 1 osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190] boot 2023-07-29T17:16:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.258339+0000 mon.a (mon.0) 502 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: audit 2023-07-29T17:16:19.259309+0000 mon.a (mon.0) 503 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.262057+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:19.263540+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.260254+0000 mon.a (mon.0) 504 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.249766+0000 mon.a (mon.0) 500 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.256090+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.256823+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.258227+0000 mon.a (mon.0) 501 : cluster 1 osd.3 [v2:172.21.15.78:6826/3632489190,v1:172.21.15.78:6827/3632489190] boot 2023-07-29T17:16:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.258339+0000 mon.a (mon.0) 502 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2023-07-29T17:16:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: audit 2023-07-29T17:16:19.259309+0000 mon.a (mon.0) 503 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T17:16:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.262057+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:19.263540+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e97 with expected crc 2023-07-29T17:16:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:20 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.260254+0000 mon.a (mon.0) 504 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2023-07-29T17:16:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.258917+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.263331+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.263348+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.264967+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.265484+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.267059+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:20.267168+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:21 smithi129 bash[53756]: cluster 2023-07-29T17:16:21.168395+0000 mgr.y (mgr.25169) 218 : cluster 0 pgmap v144: 161 pgs: 8 peering, 20 active+undersized+degraded, 38 active+undersized, 95 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 110/999 objects degraded (11.011%) 2023-07-29T17:16:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.258917+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.263331+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.263348+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.264967+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.265484+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.267059+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:20.267168+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[58614]: cluster 2023-07-29T17:16:21.168395+0000 mgr.y (mgr.25169) 218 : cluster 0 pgmap v144: 161 pgs: 8 peering, 20 active+undersized+degraded, 38 active+undersized, 95 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 110/999 objects degraded (11.011%) 2023-07-29T17:16:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.258917+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.263331+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.263348+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.264967+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.265484+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.267059+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:20.267168+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e98 with expected crc 2023-07-29T17:16:21.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:21 smithi078 bash[60360]: cluster 2023-07-29T17:16:21.168395+0000 mgr.y (mgr.25169) 218 : cluster 0 pgmap v144: 161 pgs: 8 peering, 20 active+undersized+degraded, 38 active+undersized, 95 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 110/999 objects degraded (11.011%) 2023-07-29T17:16:22.589 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:16:22.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:22 smithi129 bash[53756]: cluster 2023-07-29T17:16:21.278272+0000 mon.a (mon.0) 505 : cluster 3 Health check update: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:22 smithi078 bash[58614]: cluster 2023-07-29T17:16:21.278272+0000 mon.a (mon.0) 505 : cluster 3 Health check update: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:22.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:22 smithi078 bash[60360]: cluster 2023-07-29T17:16:21.278272+0000 mon.a (mon.0) 505 : cluster 3 Health check update: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:23.338 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:16:23.338 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6m) 5s ago 12m 10.7M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:16:23.338 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (5m) 2m ago 12m 26.1M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:16:23.338 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 2m ago 10m - - 2023-07-29T17:16:23.338 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (4m) 2m ago 16m 428M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:16:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (5m) 5s ago 18m 473M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:16:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4m) 5s ago 18m 64.3M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:16:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (2m) 2m ago 17m 37.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:16:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3m) 5s ago 17m 48.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:16:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6m) 5s ago 12m 8844k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:16:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (6m) 2m ago 12m 3644k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (99s) 5s ago 16m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (69s) 5s ago 16m 52.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (40s) 5s ago 15m 46.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10s) 5s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (14m) 2m ago 14m 55.5M 4096M 16.2.0 24ecd6d5f14c 2cda8e8daffc 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (14m) 2m ago 14m 53.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:16:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (13m) 2m ago 13m 53.3M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:16:23.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (13m) 2m ago 13m 56.5M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:16:23.341 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 2m ago 12m - - 2023-07-29T17:16:23.341 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (10m) 5s ago 10m 55.8M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:16:23.341 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (10m) 2m ago 10m 54.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:16:23.341 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (10m) 5s ago 10m 56.0M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:16:23.341 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (10m) 2m ago 10m 54.5M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:16:23.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:23 smithi129 bash[53756]: audit 2023-07-29T17:16:22.553537+0000 mgr.y (mgr.25169) 219 : audit 0 from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:23.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:23 smithi129 bash[53756]: audit 2023-07-29T17:16:22.928709+0000 mgr.y (mgr.25169) 220 : audit 0 from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:23.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:23 smithi129 bash[53756]: cluster 2023-07-29T17:16:23.169520+0000 mgr.y (mgr.25169) 221 : cluster 0 pgmap v145: 161 pgs: 8 peering, 20 active+undersized+degraded, 33 active+undersized, 100 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 110/999 objects degraded (11.011%) 2023-07-29T17:16:23.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:23 smithi078 bash[58614]: audit 2023-07-29T17:16:22.553537+0000 mgr.y (mgr.25169) 219 : audit 0 from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:23.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:23 smithi078 bash[58614]: audit 2023-07-29T17:16:22.928709+0000 mgr.y (mgr.25169) 220 : audit 0 from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:23.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:23 smithi078 bash[58614]: cluster 2023-07-29T17:16:23.169520+0000 mgr.y (mgr.25169) 221 : cluster 0 pgmap v145: 161 pgs: 8 peering, 20 active+undersized+degraded, 33 active+undersized, 100 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 110/999 objects degraded (11.011%) 2023-07-29T17:16:23.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:23 smithi078 bash[60360]: audit 2023-07-29T17:16:22.553537+0000 mgr.y (mgr.25169) 219 : audit 0 from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:23.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:23 smithi078 bash[60360]: audit 2023-07-29T17:16:22.928709+0000 mgr.y (mgr.25169) 220 : audit 0 from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:23.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:23 smithi078 bash[60360]: cluster 2023-07-29T17:16:23.169520+0000 mgr.y (mgr.25169) 221 : cluster 0 pgmap v145: 161 pgs: 8 peering, 20 active+undersized+degraded, 33 active+undersized, 100 active+clean; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 110/999 objects degraded (11.011%) 2023-07-29T17:16:23.781 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:23.782 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4, 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:16:23.783 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8, 2023-07-29T17:16:23.784 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 9 2023-07-29T17:16:23.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:16:23.784 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:16:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:16:24.150 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:16:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:16:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:16:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:16:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:16:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:16:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:16:24.152 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "9/23 daemons upgraded", 2023-07-29T17:16:24.152 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:16:24.152 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:16:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:16:24.590 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s); Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded 2023-07-29T17:16:24.590 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2023-07-29T17:16:24.591 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2023-07-29T17:16:24.592 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.9 is active+undersized+degraded, acting [4,2] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2023-07-29T17:16:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.15 is active+undersized+degraded, acting [5,7] 2023-07-29T17:16:24.594 INFO:teuthology.orchestra.run.smithi078.stdout: pg 6.c is active+undersized+degraded, acting [6,5] 2023-07-29T17:16:25.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[58614]: audit 2023-07-29T17:16:23.296712+0000 mgr.y (mgr.25169) 222 : audit 0 from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:25.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[58614]: audit 2023-07-29T17:16:23.759645+0000 mon.a (mon.0) 506 : audit 0 from='client.34494 172.21.15.78:0/2918294360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:25.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[58614]: audit 2023-07-29T17:16:24.127999+0000 mgr.y (mgr.25169) 223 : audit 0 from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:25.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[58614]: audit 2023-07-29T17:16:24.568766+0000 mon.a (mon.0) 507 : audit 0 from='client.? 172.21.15.78:0/1499273635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:16:25.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[60360]: audit 2023-07-29T17:16:23.296712+0000 mgr.y (mgr.25169) 222 : audit 0 from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:25.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[60360]: audit 2023-07-29T17:16:23.759645+0000 mon.a (mon.0) 506 : audit 0 from='client.34494 172.21.15.78:0/2918294360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:25.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[60360]: audit 2023-07-29T17:16:24.127999+0000 mgr.y (mgr.25169) 223 : audit 0 from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:25.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:24 smithi078 bash[60360]: audit 2023-07-29T17:16:24.568766+0000 mon.a (mon.0) 507 : audit 0 from='client.? 172.21.15.78:0/1499273635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:16:25.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:24 smithi129 bash[53756]: audit 2023-07-29T17:16:23.296712+0000 mgr.y (mgr.25169) 222 : audit 0 from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:25.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:24 smithi129 bash[53756]: audit 2023-07-29T17:16:23.759645+0000 mon.a (mon.0) 506 : audit 0 from='client.34494 172.21.15.78:0/2918294360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:25.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:24 smithi129 bash[53756]: audit 2023-07-29T17:16:24.127999+0000 mgr.y (mgr.25169) 223 : audit 0 from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:25.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:24 smithi129 bash[53756]: audit 2023-07-29T17:16:24.568766+0000 mon.a (mon.0) 507 : audit 0 from='client.? 172.21.15.78:0/1499273635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:16:26.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:25 smithi129 bash[53756]: cluster 2023-07-29T17:16:25.171216+0000 mgr.y (mgr.25169) 224 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 7 objects/s recovering 2023-07-29T17:16:26.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:25 smithi078 bash[58614]: cluster 2023-07-29T17:16:25.171216+0000 mgr.y (mgr.25169) 224 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 7 objects/s recovering 2023-07-29T17:16:26.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:25 smithi078 bash[60360]: cluster 2023-07-29T17:16:25.171216+0000 mgr.y (mgr.25169) 224 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 7 objects/s recovering 2023-07-29T17:16:27.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:26 smithi129 bash[53756]: cluster 2023-07-29T17:16:25.712257+0000 mon.a (mon.0) 508 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded) 2023-07-29T17:16:27.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:26 smithi078 bash[58614]: cluster 2023-07-29T17:16:25.712257+0000 mon.a (mon.0) 508 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded) 2023-07-29T17:16:27.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:26 smithi078 bash[60360]: cluster 2023-07-29T17:16:25.712257+0000 mon.a (mon.0) 508 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 110/999 objects degraded (11.011%), 20 pgs degraded) 2023-07-29T17:16:28.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:27 smithi078 bash[58614]: cluster 2023-07-29T17:16:27.172154+0000 mgr.y (mgr.25169) 225 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2023-07-29T17:16:28.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:27 smithi078 bash[60360]: cluster 2023-07-29T17:16:27.172154+0000 mgr.y (mgr.25169) 225 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2023-07-29T17:16:28.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:27 smithi129 bash[53756]: cluster 2023-07-29T17:16:27.172154+0000 mgr.y (mgr.25169) 225 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2023-07-29T17:16:29.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.408137+0000 mon.a (mon.0) 509 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.758036+0000 mon.a (mon.0) 510 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.769646+0000 mon.a (mon.0) 511 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.771651+0000 mon.a (mon.0) 512 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.773245+0000 mon.a (mon.0) 513 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:16:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: cephadm 2023-07-29T17:16:28.774612+0000 mgr.y (mgr.25169) 226 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:16:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: cluster 2023-07-29T17:16:28.776089+0000 mgr.y (mgr.25169) 227 : cluster 0 pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: cluster 2023-07-29T17:16:28.776455+0000 mgr.y (mgr.25169) 228 : cluster 0 pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.784112+0000 mon.a (mon.0) 514 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.868401+0000 mon.a (mon.0) 515 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:16:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.873690+0000 mon.a (mon.0) 516 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.877136+0000 mon.a (mon.0) 517 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.879903+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.883259+0000 mon.a (mon.0) 519 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-29T17:16:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: audit 2023-07-29T17:16:28.883662+0000 mgr.y (mgr.25169) 229 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-29T17:16:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:29 smithi129 bash[53756]: cephadm 2023-07-29T17:16:28.886134+0000 mgr.y (mgr.25169) 230 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-07-29T17:16:29.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.408137+0000 mon.a (mon.0) 509 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.758036+0000 mon.a (mon.0) 510 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.769646+0000 mon.a (mon.0) 511 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.771651+0000 mon.a (mon.0) 512 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.773245+0000 mon.a (mon.0) 513 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: cephadm 2023-07-29T17:16:28.774612+0000 mgr.y (mgr.25169) 226 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: cluster 2023-07-29T17:16:28.776089+0000 mgr.y (mgr.25169) 227 : cluster 0 pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: cluster 2023-07-29T17:16:28.776455+0000 mgr.y (mgr.25169) 228 : cluster 0 pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.784112+0000 mon.a (mon.0) 514 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.868401+0000 mon.a (mon.0) 515 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:16:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.873690+0000 mon.a (mon.0) 516 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.877136+0000 mon.a (mon.0) 517 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.879903+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.883259+0000 mon.a (mon.0) 519 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-29T17:16:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: audit 2023-07-29T17:16:28.883662+0000 mgr.y (mgr.25169) 229 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-29T17:16:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[58614]: cephadm 2023-07-29T17:16:28.886134+0000 mgr.y (mgr.25169) 230 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-07-29T17:16:29.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.408137+0000 mon.a (mon.0) 509 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.758036+0000 mon.a (mon.0) 510 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.769646+0000 mon.a (mon.0) 511 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.771651+0000 mon.a (mon.0) 512 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.773245+0000 mon.a (mon.0) 513 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: cephadm 2023-07-29T17:16:28.774612+0000 mgr.y (mgr.25169) 226 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: cluster 2023-07-29T17:16:28.776089+0000 mgr.y (mgr.25169) 227 : cluster 0 pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: cluster 2023-07-29T17:16:28.776455+0000 mgr.y (mgr.25169) 228 : cluster 0 pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.784112+0000 mon.a (mon.0) 514 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.868401+0000 mon.a (mon.0) 515 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.873690+0000 mon.a (mon.0) 516 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.877136+0000 mon.a (mon.0) 517 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.879903+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.883259+0000 mon.a (mon.0) 519 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: audit 2023-07-29T17:16:28.883662+0000 mgr.y (mgr.25169) 229 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-29T17:16:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:29 smithi078 bash[60360]: cephadm 2023-07-29T17:16:28.886134+0000 mgr.y (mgr.25169) 230 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-07-29T17:16:32.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:31 smithi129 bash[53756]: cluster 2023-07-29T17:16:30.777924+0000 mgr.y (mgr.25169) 231 : cluster 0 pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:32.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:31 smithi129 bash[53756]: audit 2023-07-29T17:16:31.717919+0000 mon.a (mon.0) 520 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:32.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:31 smithi129 bash[53756]: audit 2023-07-29T17:16:31.719592+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:16:32.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:31 smithi129 bash[53756]: audit 2023-07-29T17:16:31.721064+0000 mon.a (mon.0) 522 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:32.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[58614]: cluster 2023-07-29T17:16:30.777924+0000 mgr.y (mgr.25169) 231 : cluster 0 pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:32.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[58614]: audit 2023-07-29T17:16:31.717919+0000 mon.a (mon.0) 520 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:32.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[58614]: audit 2023-07-29T17:16:31.719592+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:16:32.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[58614]: audit 2023-07-29T17:16:31.721064+0000 mon.a (mon.0) 522 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:32.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[60360]: cluster 2023-07-29T17:16:30.777924+0000 mgr.y (mgr.25169) 231 : cluster 0 pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:32.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[60360]: audit 2023-07-29T17:16:31.717919+0000 mon.a (mon.0) 520 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:32.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[60360]: audit 2023-07-29T17:16:31.719592+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T17:16:32.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:31 smithi078 bash[60360]: audit 2023-07-29T17:16:31.721064+0000 mon.a (mon.0) 522 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:16:33.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:32 smithi129 bash[53756]: cephadm 2023-07-29T17:16:31.708081+0000 mgr.y (mgr.25169) 232 : cephadm 1 Upgrade: Updating osd.4 2023-07-29T17:16:33.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:32 smithi129 bash[53756]: cephadm 2023-07-29T17:16:31.726098+0000 mgr.y (mgr.25169) 233 : cephadm 1 Deploying daemon osd.4 on smithi129 2023-07-29T17:16:33.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:32 smithi078 bash[58614]: cephadm 2023-07-29T17:16:31.708081+0000 mgr.y (mgr.25169) 232 : cephadm 1 Upgrade: Updating osd.4 2023-07-29T17:16:33.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:32 smithi078 bash[58614]: cephadm 2023-07-29T17:16:31.726098+0000 mgr.y (mgr.25169) 233 : cephadm 1 Deploying daemon osd.4 on smithi129 2023-07-29T17:16:33.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:32 smithi078 bash[60360]: cephadm 2023-07-29T17:16:31.708081+0000 mgr.y (mgr.25169) 232 : cephadm 1 Upgrade: Updating osd.4 2023-07-29T17:16:33.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:32 smithi078 bash[60360]: cephadm 2023-07-29T17:16:31.726098+0000 mgr.y (mgr.25169) 233 : cephadm 1 Deploying daemon osd.4 on smithi129 2023-07-29T17:16:34.124 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:33 smithi129 bash[53756]: cluster 2023-07-29T17:16:32.778809+0000 mgr.y (mgr.25169) 234 : cluster 0 pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:34.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:33 smithi078 bash[58614]: cluster 2023-07-29T17:16:32.778809+0000 mgr.y (mgr.25169) 234 : cluster 0 pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:34.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:33 smithi078 bash[60360]: cluster 2023-07-29T17:16:32.778809+0000 mgr.y (mgr.25169) 234 : cluster 0 pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2023-07-29T17:16:36.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:35 smithi129 bash[53756]: cluster 2023-07-29T17:16:34.780465+0000 mgr.y (mgr.25169) 235 : cluster 0 pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:36.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:35 smithi078 bash[58614]: cluster 2023-07-29T17:16:34.780465+0000 mgr.y (mgr.25169) 235 : cluster 0 pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:36.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:35 smithi078 bash[60360]: cluster 2023-07-29T17:16:34.780465+0000 mgr.y (mgr.25169) 235 : cluster 0 pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:36.674 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:36 smithi129 systemd[1]: Stopping Ceph osd.4 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:16:37.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:36 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.758544+0000 mon.a (mon.0) 523 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-29T17:16:37.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:36 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.758617+0000 mon.a (mon.0) 524 : cluster 1 osd.4 failed (root=default,host=smithi129) (connection refused reported by osd.3) 2023-07-29T17:16:37.174 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:36 smithi129 bash[26703]: debug 2023-07-29T17:16:36.685+0000 7f3f4feb6700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:16:37.174 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:36 smithi129 bash[26703]: debug 2023-07-29T17:16:36.685+0000 7f3f4feb6700 -1 osd.4 98 *** Got signal Terminated *** 2023-07-29T17:16:37.175 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:36 smithi129 bash[26703]: debug 2023-07-29T17:16:36.685+0000 7f3f4feb6700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:16:37.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:36 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.758544+0000 mon.a (mon.0) 523 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-29T17:16:37.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:36 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.758617+0000 mon.a (mon.0) 524 : cluster 1 osd.4 failed (root=default,host=smithi129) (connection refused reported by osd.3) 2023-07-29T17:16:37.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:36 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.758544+0000 mon.a (mon.0) 523 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-29T17:16:37.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:36 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.758617+0000 mon.a (mon.0) 524 : cluster 1 osd.4 failed (root=default,host=smithi129) (connection refused reported by osd.3) 2023-07-29T17:16:37.881 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[55857]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.4 2023-07-29T17:16:38.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.781205+0000 mgr.y (mgr.25169) 236 : cluster 0 pgmap v153: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:38.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.855868+0000 mon.a (mon.0) 525 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:16:38.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.872851+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.876192+0000 mon.a (mon.0) 526 : cluster 0 osdmap e99: 8 total, 7 up, 8 in 2023-07-29T17:16:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.877242+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.880675+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:37 smithi129 bash[53756]: cluster 2023-07-29T17:16:36.883938+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.781205+0000 mgr.y (mgr.25169) 236 : cluster 0 pgmap v153: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.855868+0000 mon.a (mon.0) 525 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:16:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.872851+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.876192+0000 mon.a (mon.0) 526 : cluster 0 osdmap e99: 8 total, 7 up, 8 in 2023-07-29T17:16:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.877242+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.880675+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[58614]: cluster 2023-07-29T17:16:36.883938+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.781205+0000 mgr.y (mgr.25169) 236 : cluster 0 pgmap v153: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.855868+0000 mon.a (mon.0) 525 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.872851+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.876192+0000 mon.a (mon.0) 526 : cluster 0 osdmap e99: 8 total, 7 up, 8 in 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.877242+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.880675+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:38.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:37 smithi078 bash[60360]: cluster 2023-07-29T17:16:36.883938+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e99 with expected crc 2023-07-29T17:16:39.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.873698+0000 mon.a (mon.0) 527 : cluster 0 osdmap e100: 8 total, 7 up, 8 in 2023-07-29T17:16:39.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.875005+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.875654+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.877398+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.878144+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.878536+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.881478+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.884488+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.887073+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:38 smithi129 bash[53756]: cluster 2023-07-29T17:16:37.887324+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.873698+0000 mon.a (mon.0) 527 : cluster 0 osdmap e100: 8 total, 7 up, 8 in 2023-07-29T17:16:39.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.875005+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.875654+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.877398+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.878144+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.878536+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.881478+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.884488+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.887073+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[58614]: cluster 2023-07-29T17:16:37.887324+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.873698+0000 mon.a (mon.0) 527 : cluster 0 osdmap e100: 8 total, 7 up, 8 in 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.875005+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.875654+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.877398+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.878144+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.878536+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.881478+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.884488+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.887073+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:38 smithi078 bash[60360]: cluster 2023-07-29T17:16:37.887324+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e100 with expected crc 2023-07-29T17:16:40.174 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:39 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.4.service: Succeeded. 2023-07-29T17:16:40.174 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:39 smithi129 systemd[1]: Stopped Ceph osd.4 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:16:40.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:39 smithi129 bash[53756]: cluster 2023-07-29T17:16:38.782264+0000 mgr.y (mgr.25169) 237 : cluster 0 pgmap v156: 161 pgs: 1 active+undersized+degraded, 7 peering, 13 stale+active+clean, 9 active+undersized, 131 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 9/999 objects degraded (0.901%) 2023-07-29T17:16:40.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:39 smithi129 bash[53756]: cluster 2023-07-29T17:16:39.714672+0000 mon.a (mon.0) 528 : cluster 3 Health check failed: Degraded data redundancy: 9/999 objects degraded (0.901%), 1 pg degraded (PG_DEGRADED) 2023-07-29T17:16:40.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:39 smithi078 bash[58614]: cluster 2023-07-29T17:16:38.782264+0000 mgr.y (mgr.25169) 237 : cluster 0 pgmap v156: 161 pgs: 1 active+undersized+degraded, 7 peering, 13 stale+active+clean, 9 active+undersized, 131 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 9/999 objects degraded (0.901%) 2023-07-29T17:16:40.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:39 smithi078 bash[58614]: cluster 2023-07-29T17:16:39.714672+0000 mon.a (mon.0) 528 : cluster 3 Health check failed: Degraded data redundancy: 9/999 objects degraded (0.901%), 1 pg degraded (PG_DEGRADED) 2023-07-29T17:16:40.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:39 smithi078 bash[60360]: cluster 2023-07-29T17:16:38.782264+0000 mgr.y (mgr.25169) 237 : cluster 0 pgmap v156: 161 pgs: 1 active+undersized+degraded, 7 peering, 13 stale+active+clean, 9 active+undersized, 131 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 9/999 objects degraded (0.901%) 2023-07-29T17:16:40.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:39 smithi078 bash[60360]: cluster 2023-07-29T17:16:39.714672+0000 mon.a (mon.0) 528 : cluster 3 Health check failed: Degraded data redundancy: 9/999 objects degraded (0.901%), 1 pg degraded (PG_DEGRADED) 2023-07-29T17:16:40.674 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:40 smithi129 systemd[1]: Started Ceph osd.4 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:16:41.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[53756]: audit 2023-07-29T17:16:40.270567+0000 mon.a (mon.0) 529 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:41.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[53756]: audit 2023-07-29T17:16:40.280878+0000 mon.a (mon.0) 530 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:41.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[53756]: cluster 2023-07-29T17:16:40.784061+0000 mgr.y (mgr.25169) 238 : cluster 0 pgmap v157: 161 pgs: 23 active+undersized+degraded, 7 peering, 38 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 155/999 objects degraded (15.516%) 2023-07-29T17:16:41.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:41 smithi078 bash[60360]: audit 2023-07-29T17:16:40.270567+0000 mon.a (mon.0) 529 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:41.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:41 smithi078 bash[60360]: audit 2023-07-29T17:16:40.280878+0000 mon.a (mon.0) 530 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:41.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:41 smithi078 bash[60360]: cluster 2023-07-29T17:16:40.784061+0000 mgr.y (mgr.25169) 238 : cluster 0 pgmap v157: 161 pgs: 23 active+undersized+degraded, 7 peering, 38 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 155/999 objects degraded (15.516%) 2023-07-29T17:16:41.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:41 smithi078 bash[58614]: audit 2023-07-29T17:16:40.270567+0000 mon.a (mon.0) 529 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:41.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:41 smithi078 bash[58614]: audit 2023-07-29T17:16:40.280878+0000 mon.a (mon.0) 530 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:41.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:41 smithi078 bash[58614]: cluster 2023-07-29T17:16:40.784061+0000 mgr.y (mgr.25169) 238 : cluster 0 pgmap v157: 161 pgs: 23 active+undersized+degraded, 7 peering, 38 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 155/999 objects degraded (15.516%) 2023-07-29T17:16:42.424 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-29T17:16:42.425 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-07-29T17:16:42.425 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-07-29T17:16:42.425 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-29T17:16:42.425 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2023-07-29T17:16:42.425 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-29T17:16:42.425 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:41 smithi129 bash[56121]: --> ceph-volume raw activate successful for osd ID: 4 2023-07-29T17:16:44.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:43 smithi129 bash[53756]: cluster 2023-07-29T17:16:42.785282+0000 mgr.y (mgr.25169) 239 : cluster 0 pgmap v158: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:44.174 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:43 smithi129 bash[56428]: debug 2023-07-29T17:16:43.809+0000 7fbca2591700 1 -- 172.21.15.129:0/3141908738 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564faafa24e0 con 0x564faaffd000 2023-07-29T17:16:44.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:43 smithi078 bash[58614]: cluster 2023-07-29T17:16:42.785282+0000 mgr.y (mgr.25169) 239 : cluster 0 pgmap v158: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:44.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:43 smithi078 bash[60360]: cluster 2023-07-29T17:16:42.785282+0000 mgr.y (mgr.25169) 239 : cluster 0 pgmap v158: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:44.674 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:44 smithi129 bash[56428]: debug 2023-07-29T17:16:44.357+0000 7fbcab716640 -1 Falling back to public interface 2023-07-29T17:16:46.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:45 smithi129 bash[53756]: cluster 2023-07-29T17:16:44.787146+0000 mgr.y (mgr.25169) 240 : cluster 0 pgmap v159: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:45 smithi078 bash[58614]: cluster 2023-07-29T17:16:44.787146+0000 mgr.y (mgr.25169) 240 : cluster 0 pgmap v159: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:46.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:45 smithi078 bash[60360]: cluster 2023-07-29T17:16:44.787146+0000 mgr.y (mgr.25169) 240 : cluster 0 pgmap v159: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:47.887 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:47 smithi129 bash[56428]: debug 2023-07-29T17:16:47.501+0000 7fbcab716640 -1 osd.4 98 log_to_monitors true 2023-07-29T17:16:47.887 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:47 smithi129 bash[53756]: cluster 2023-07-29T17:16:46.788201+0000 mgr.y (mgr.25169) 241 : cluster 0 pgmap v160: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:47.887 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:47 smithi129 bash[53756]: audit 2023-07-29T17:16:47.508559+0000 mon.a (mon.0) 531 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:16:47.887 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:47 smithi129 bash[53756]: audit 2023-07-29T17:16:47.509949+0000 mon.b (mon.1) 3 : audit 1 from='osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:16:48.174 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:16:47 smithi129 bash[56428]: debug 2023-07-29T17:16:47.917+0000 7fbc9ab7f700 -1 osd.4 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:16:48.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:47 smithi078 bash[58614]: cluster 2023-07-29T17:16:46.788201+0000 mgr.y (mgr.25169) 241 : cluster 0 pgmap v160: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:48.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:47 smithi078 bash[58614]: audit 2023-07-29T17:16:47.508559+0000 mon.a (mon.0) 531 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:16:48.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:47 smithi078 bash[58614]: audit 2023-07-29T17:16:47.509949+0000 mon.b (mon.1) 3 : audit 1 from='osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:16:48.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:47 smithi078 bash[60360]: cluster 2023-07-29T17:16:46.788201+0000 mgr.y (mgr.25169) 241 : cluster 0 pgmap v160: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:48.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:47 smithi078 bash[60360]: audit 2023-07-29T17:16:47.508559+0000 mon.a (mon.0) 531 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:16:48.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:47 smithi078 bash[60360]: audit 2023-07-29T17:16:47.509949+0000 mon.b (mon.1) 3 : audit 1 from='osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T17:16:49.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: cluster 2023-07-29T17:16:47.855644+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e101 with expected crc 2023-07-29T17:16:49.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: cluster 2023-07-29T17:16:47.856627+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e101 with expected crc 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:47.857551+0000 mon.a (mon.0) 532 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: cluster 2023-07-29T17:16:47.857736+0000 mon.a (mon.0) 533 : cluster 0 osdmap e101: 8 total, 7 up, 8 in 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:47.865708+0000 mon.b (mon.1) 4 : audit 1 from='osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:47.870564+0000 mon.a (mon.0) 534 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:47.874958+0000 mon.a (mon.0) 535 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:47.883950+0000 mon.a (mon.0) 536 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:48.438010+0000 mon.a (mon.0) 537 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:48.632238+0000 mon.a (mon.0) 538 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: audit 2023-07-29T17:16:48.643564+0000 mon.a (mon.0) 539 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:48 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.714821+0000 mon.a (mon.0) 540 : cluster 3 Health check update: Degraded data redundancy: 165/999 objects degraded (16.517%), 26 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: cluster 2023-07-29T17:16:47.855644+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e101 with expected crc 2023-07-29T17:16:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: cluster 2023-07-29T17:16:47.856627+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e101 with expected crc 2023-07-29T17:16:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:47.857551+0000 mon.a (mon.0) 532 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: cluster 2023-07-29T17:16:47.857736+0000 mon.a (mon.0) 533 : cluster 0 osdmap e101: 8 total, 7 up, 8 in 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:47.865708+0000 mon.b (mon.1) 4 : audit 1 from='osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:47.870564+0000 mon.a (mon.0) 534 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:47.874958+0000 mon.a (mon.0) 535 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:47.883950+0000 mon.a (mon.0) 536 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:48.438010+0000 mon.a (mon.0) 537 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:48.632238+0000 mon.a (mon.0) 538 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: audit 2023-07-29T17:16:48.643564+0000 mon.a (mon.0) 539 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.714821+0000 mon.a (mon.0) 540 : cluster 3 Health check update: Degraded data redundancy: 165/999 objects degraded (16.517%), 26 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: cluster 2023-07-29T17:16:47.855644+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e101 with expected crc 2023-07-29T17:16:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: cluster 2023-07-29T17:16:47.856627+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e101 with expected crc 2023-07-29T17:16:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:47.857551+0000 mon.a (mon.0) 532 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T17:16:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: cluster 2023-07-29T17:16:47.857736+0000 mon.a (mon.0) 533 : cluster 0 osdmap e101: 8 total, 7 up, 8 in 2023-07-29T17:16:49.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:47.865708+0000 mon.b (mon.1) 4 : audit 1 from='osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:47.870564+0000 mon.a (mon.0) 534 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:47.874958+0000 mon.a (mon.0) 535 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:47.883950+0000 mon.a (mon.0) 536 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:48.438010+0000 mon.a (mon.0) 537 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:48.632238+0000 mon.a (mon.0) 538 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: audit 2023-07-29T17:16:48.643564+0000 mon.a (mon.0) 539 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:16:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:48 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.714821+0000 mon.a (mon.0) 540 : cluster 3 Health check update: Degraded data redundancy: 165/999 objects degraded (16.517%), 26 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.789549+0000 mgr.y (mgr.25169) 242 : cluster 0 pgmap v162: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.881195+0000 mon.a (mon.0) 541 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:16:50.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.889024+0000 mon.a (mon.0) 542 : cluster 1 osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120] boot 2023-07-29T17:16:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.889085+0000 mon.a (mon.0) 543 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-07-29T17:16:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.889128+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: audit 2023-07-29T17:16:48.889446+0000 mon.a (mon.0) 544 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:16:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.891982+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:49 smithi129 bash[53756]: cluster 2023-07-29T17:16:48.892933+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.789549+0000 mgr.y (mgr.25169) 242 : cluster 0 pgmap v162: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:50.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.881195+0000 mon.a (mon.0) 541 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:16:50.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.889024+0000 mon.a (mon.0) 542 : cluster 1 osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120] boot 2023-07-29T17:16:50.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.889085+0000 mon.a (mon.0) 543 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.889128+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: audit 2023-07-29T17:16:48.889446+0000 mon.a (mon.0) 544 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.891982+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[58614]: cluster 2023-07-29T17:16:48.892933+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.789549+0000 mgr.y (mgr.25169) 242 : cluster 0 pgmap v162: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 165/999 objects degraded (16.517%) 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.881195+0000 mon.a (mon.0) 541 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:16:50.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.889024+0000 mon.a (mon.0) 542 : cluster 1 osd.4 [v2:172.21.15.129:6800/420080120,v1:172.21.15.129:6801/420080120] boot 2023-07-29T17:16:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.889085+0000 mon.a (mon.0) 543 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-07-29T17:16:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.889128+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: audit 2023-07-29T17:16:48.889446+0000 mon.a (mon.0) 544 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T17:16:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.891982+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:49 smithi078 bash[60360]: cluster 2023-07-29T17:16:48.892933+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e102 with expected crc 2023-07-29T17:16:51.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.890868+0000 mon.a (mon.0) 545 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-07-29T17:16:51.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.891240+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.891569+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.893896+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.895563+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.898456+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.898569+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.901109+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:50 smithi129 bash[53756]: cluster 2023-07-29T17:16:49.903905+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.890868+0000 mon.a (mon.0) 545 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-07-29T17:16:51.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.891240+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.891569+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.893896+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.895563+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.898456+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.898569+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.901109+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[58614]: cluster 2023-07-29T17:16:49.903905+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.890868+0000 mon.a (mon.0) 545 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.891240+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.891569+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.893896+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.895563+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.898456+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.898569+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.901109+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:51.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:50 smithi078 bash[60360]: cluster 2023-07-29T17:16:49.903905+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e103 with expected crc 2023-07-29T17:16:52.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:51 smithi129 bash[53756]: cluster 2023-07-29T17:16:50.790491+0000 mgr.y (mgr.25169) 243 : cluster 0 pgmap v165: 161 pgs: 38 peering, 10 active+undersized+degraded, 20 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:16:52.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:51 smithi078 bash[60360]: cluster 2023-07-29T17:16:50.790491+0000 mgr.y (mgr.25169) 243 : cluster 0 pgmap v165: 161 pgs: 38 peering, 10 active+undersized+degraded, 20 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:16:52.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:51 smithi078 bash[58614]: cluster 2023-07-29T17:16:50.790491+0000 mgr.y (mgr.25169) 243 : cluster 0 pgmap v165: 161 pgs: 38 peering, 10 active+undersized+degraded, 20 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:16:54.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:53 smithi078 bash[58614]: cluster 2023-07-29T17:16:52.791538+0000 mgr.y (mgr.25169) 244 : cluster 0 pgmap v166: 161 pgs: 38 peering, 10 active+undersized+degraded, 17 active+undersized, 96 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:16:54.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:53 smithi078 bash[58614]: cluster 2023-07-29T17:16:53.716387+0000 mon.a (mon.0) 546 : cluster 3 Health check update: Degraded data redundancy: 49/999 objects degraded (4.905%), 10 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:54.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:53 smithi078 bash[60360]: cluster 2023-07-29T17:16:52.791538+0000 mgr.y (mgr.25169) 244 : cluster 0 pgmap v166: 161 pgs: 38 peering, 10 active+undersized+degraded, 17 active+undersized, 96 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:16:54.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:53 smithi078 bash[60360]: cluster 2023-07-29T17:16:53.716387+0000 mon.a (mon.0) 546 : cluster 3 Health check update: Degraded data redundancy: 49/999 objects degraded (4.905%), 10 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:54.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:53 smithi129 bash[53756]: cluster 2023-07-29T17:16:52.791538+0000 mgr.y (mgr.25169) 244 : cluster 0 pgmap v166: 161 pgs: 38 peering, 10 active+undersized+degraded, 17 active+undersized, 96 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:16:54.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:53 smithi129 bash[53756]: cluster 2023-07-29T17:16:53.716387+0000 mon.a (mon.0) 546 : cluster 3 Health check update: Degraded data redundancy: 49/999 objects degraded (4.905%), 10 pgs degraded (PG_DEGRADED) 2023-07-29T17:16:54.983 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:16:55.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:54 smithi129 bash[53756]: cluster 2023-07-29T17:16:54.907387+0000 mon.a (mon.0) 547 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/999 objects degraded (4.905%), 10 pgs degraded) 2023-07-29T17:16:55.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:54 smithi078 bash[58614]: cluster 2023-07-29T17:16:54.907387+0000 mon.a (mon.0) 547 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/999 objects degraded (4.905%), 10 pgs degraded) 2023-07-29T17:16:55.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:54 smithi078 bash[60360]: cluster 2023-07-29T17:16:54.907387+0000 mon.a (mon.0) 547 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/999 objects degraded (4.905%), 10 pgs degraded) 2023-07-29T17:16:55.717 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:16:55.717 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6m) 37s ago 13m 10.7M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:16:55.717 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (6m) 7s ago 12m 26.4M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:16:55.718 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 7s ago 10m - - 2023-07-29T17:16:55.718 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (4m) 7s ago 17m 429M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:16:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (5m) 37s ago 19m 473M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:16:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4m) 37s ago 19m 64.3M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:16:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (3m) 7s ago 17m 46.8M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:16:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3m) 37s ago 17m 48.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:16:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7m) 37s ago 13m 8844k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:16:55.719 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (6m) 7s ago 13m 3728k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2m) 37s ago 17m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (102s) 37s ago 16m 52.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (72s) 37s ago 16m 46.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (42s) 37s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (12s) 7s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (14m) 7s ago 14m 61.1M 4096M 16.2.0 24ecd6d5f14c a93906025df2 2023-07-29T17:16:55.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (14m) 7s ago 14m 60.6M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:16:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (14m) 7s ago 14m 64.2M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:16:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 7s ago 13m - - 2023-07-29T17:16:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (11m) 37s ago 11m 55.8M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:16:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (11m) 7s ago 11m 55.9M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:16:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (10m) 37s ago 10m 56.0M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:16:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (10m) 7s ago 10m 55.9M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:16:56.160 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:16:56.160 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:16:56.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:16:56.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:56.160 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3, 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 5 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:56.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:16:56.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:16:56.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:16:56.162 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:16:56.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2023-07-29T17:16:56.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 10 2023-07-29T17:16:56.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:16:56.163 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:16:56.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:55 smithi129 bash[53756]: cluster 2023-07-29T17:16:54.793230+0000 mgr.y (mgr.25169) 245 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:56.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:55 smithi129 bash[53756]: audit 2023-07-29T17:16:54.945835+0000 mgr.y (mgr.25169) 246 : audit 0 from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:56.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:55 smithi078 bash[58614]: cluster 2023-07-29T17:16:54.793230+0000 mgr.y (mgr.25169) 245 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:56.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:55 smithi078 bash[58614]: audit 2023-07-29T17:16:54.945835+0000 mgr.y (mgr.25169) 246 : audit 0 from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:56.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:55 smithi078 bash[60360]: cluster 2023-07-29T17:16:54.793230+0000 mgr.y (mgr.25169) 245 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:56.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:55 smithi078 bash[60360]: audit 2023-07-29T17:16:54.945835+0000 mgr.y (mgr.25169) 246 : audit 0 from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:56.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:16:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:16:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:16:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:16:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:16:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:16:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:16:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:16:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "10/23 daemons upgraded", 2023-07-29T17:16:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:16:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:16:56.538 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:16:56.947 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:16:56.947 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:16:56.947 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:16:56.948 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:16:57.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:56 smithi129 bash[53756]: audit 2023-07-29T17:16:55.315587+0000 mgr.y (mgr.25169) 247 : audit 0 from='client.44354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:57.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:56 smithi129 bash[53756]: audit 2023-07-29T17:16:55.674227+0000 mgr.y (mgr.25169) 248 : audit 0 from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:57.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:56 smithi129 bash[53756]: audit 2023-07-29T17:16:56.137576+0000 mon.a (mon.0) 548 : audit 0 from='client.34536 172.21.15.78:0/1786246633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:57.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:56 smithi129 bash[53756]: audit 2023-07-29T17:16:56.925581+0000 mon.a (mon.0) 549 : audit 0 from='client.34548 172.21.15.78:0/84381908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:16:57.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[58614]: audit 2023-07-29T17:16:55.315587+0000 mgr.y (mgr.25169) 247 : audit 0 from='client.44354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:57.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[58614]: audit 2023-07-29T17:16:55.674227+0000 mgr.y (mgr.25169) 248 : audit 0 from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:57.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[58614]: audit 2023-07-29T17:16:56.137576+0000 mon.a (mon.0) 548 : audit 0 from='client.34536 172.21.15.78:0/1786246633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:57.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[58614]: audit 2023-07-29T17:16:56.925581+0000 mon.a (mon.0) 549 : audit 0 from='client.34548 172.21.15.78:0/84381908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:16:57.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[60360]: audit 2023-07-29T17:16:55.315587+0000 mgr.y (mgr.25169) 247 : audit 0 from='client.44354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:57.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[60360]: audit 2023-07-29T17:16:55.674227+0000 mgr.y (mgr.25169) 248 : audit 0 from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:57.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[60360]: audit 2023-07-29T17:16:56.137576+0000 mon.a (mon.0) 548 : audit 0 from='client.34536 172.21.15.78:0/1786246633' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:16:57.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:56 smithi078 bash[60360]: audit 2023-07-29T17:16:56.925581+0000 mon.a (mon.0) 549 : audit 0 from='client.34548 172.21.15.78:0/84381908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:16:58.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:57 smithi129 bash[53756]: audit 2023-07-29T17:16:56.514230+0000 mgr.y (mgr.25169) 249 : audit 0 from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:58.259 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:57 smithi129 bash[53756]: cluster 2023-07-29T17:16:56.793966+0000 mgr.y (mgr.25169) 250 : cluster 0 pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:58.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:57 smithi078 bash[58614]: audit 2023-07-29T17:16:56.514230+0000 mgr.y (mgr.25169) 249 : audit 0 from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:58.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:57 smithi078 bash[58614]: cluster 2023-07-29T17:16:56.793966+0000 mgr.y (mgr.25169) 250 : cluster 0 pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:16:58.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:57 smithi078 bash[60360]: audit 2023-07-29T17:16:56.514230+0000 mgr.y (mgr.25169) 249 : audit 0 from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:16:58.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:57 smithi078 bash[60360]: cluster 2023-07-29T17:16:56.793966+0000 mgr.y (mgr.25169) 250 : cluster 0 pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.361 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: cluster 2023-07-29T17:16:58.795122+0000 mgr.y (mgr.25169) 251 : cluster 0 pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.361 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:58.952519+0000 mon.a (mon.0) 550 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.361 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:58.964092+0000 mon.a (mon.0) 551 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.361 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:58.965950+0000 mon.a (mon.0) 552 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:58.967398+0000 mon.a (mon.0) 553 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: cephadm 2023-07-29T17:16:58.968579+0000 mgr.y (mgr.25169) 252 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: cluster 2023-07-29T17:16:58.969943+0000 mgr.y (mgr.25169) 253 : cluster 0 pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: cluster 2023-07-29T17:16:58.970579+0000 mgr.y (mgr.25169) 254 : cluster 0 pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:58.978566+0000 mon.a (mon.0) 554 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:59.067262+0000 mon.a (mon.0) 555 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:00.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:59.072946+0000 mon.a (mon.0) 556 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:59.076703+0000 mon.a (mon.0) 557 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:59.079477+0000 mon.a (mon.0) 558 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:59.082087+0000 mon.a (mon.0) 559 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-29T17:17:00.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: audit 2023-07-29T17:16:59.082373+0000 mgr.y (mgr.25169) 255 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-29T17:17:00.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:16:59 smithi129 bash[53756]: cephadm 2023-07-29T17:16:59.084150+0000 mgr.y (mgr.25169) 256 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-07-29T17:17:00.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: cluster 2023-07-29T17:16:58.795122+0000 mgr.y (mgr.25169) 251 : cluster 0 pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:58.952519+0000 mon.a (mon.0) 550 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:58.964092+0000 mon.a (mon.0) 551 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:58.965950+0000 mon.a (mon.0) 552 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:58.967398+0000 mon.a (mon.0) 553 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: cephadm 2023-07-29T17:16:58.968579+0000 mgr.y (mgr.25169) 252 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: cluster 2023-07-29T17:16:58.969943+0000 mgr.y (mgr.25169) 253 : cluster 0 pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: cluster 2023-07-29T17:16:58.970579+0000 mgr.y (mgr.25169) 254 : cluster 0 pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:58.978566+0000 mon.a (mon.0) 554 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:59.067262+0000 mon.a (mon.0) 555 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:59.072946+0000 mon.a (mon.0) 556 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:59.076703+0000 mon.a (mon.0) 557 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:59.079477+0000 mon.a (mon.0) 558 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:59.082087+0000 mon.a (mon.0) 559 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-29T17:17:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: audit 2023-07-29T17:16:59.082373+0000 mgr.y (mgr.25169) 255 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-29T17:17:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[58614]: cephadm 2023-07-29T17:16:59.084150+0000 mgr.y (mgr.25169) 256 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-07-29T17:17:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: cluster 2023-07-29T17:16:58.795122+0000 mgr.y (mgr.25169) 251 : cluster 0 pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:58.952519+0000 mon.a (mon.0) 550 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:58.964092+0000 mon.a (mon.0) 551 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:58.965950+0000 mon.a (mon.0) 552 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:58.967398+0000 mon.a (mon.0) 553 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: cephadm 2023-07-29T17:16:58.968579+0000 mgr.y (mgr.25169) 252 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: cluster 2023-07-29T17:16:58.969943+0000 mgr.y (mgr.25169) 253 : cluster 0 pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: cluster 2023-07-29T17:16:58.970579+0000 mgr.y (mgr.25169) 254 : cluster 0 pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:58.978566+0000 mon.a (mon.0) 554 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:59.067262+0000 mon.a (mon.0) 555 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:59.072946+0000 mon.a (mon.0) 556 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:59.076703+0000 mon.a (mon.0) 557 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:59.079477+0000 mon.a (mon.0) 558 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:00.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:59.082087+0000 mon.a (mon.0) 559 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-29T17:17:00.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: audit 2023-07-29T17:16:59.082373+0000 mgr.y (mgr.25169) 255 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-29T17:17:00.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:16:59 smithi078 bash[60360]: cephadm 2023-07-29T17:16:59.084150+0000 mgr.y (mgr.25169) 256 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-07-29T17:17:02.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[58614]: cluster 2023-07-29T17:17:00.971918+0000 mgr.y (mgr.25169) 257 : cluster 0 pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:02.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[58614]: audit 2023-07-29T17:17:01.314263+0000 mon.a (mon.0) 560 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:02.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[58614]: audit 2023-07-29T17:17:01.316496+0000 mon.a (mon.0) 561 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:17:02.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[58614]: audit 2023-07-29T17:17:01.318288+0000 mon.a (mon.0) 562 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:02.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[60360]: cluster 2023-07-29T17:17:00.971918+0000 mgr.y (mgr.25169) 257 : cluster 0 pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:02.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[60360]: audit 2023-07-29T17:17:01.314263+0000 mon.a (mon.0) 560 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:02.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[60360]: audit 2023-07-29T17:17:01.316496+0000 mon.a (mon.0) 561 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:17:02.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:02 smithi078 bash[60360]: audit 2023-07-29T17:17:01.318288+0000 mon.a (mon.0) 562 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:02.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:02 smithi129 bash[53756]: cluster 2023-07-29T17:17:00.971918+0000 mgr.y (mgr.25169) 257 : cluster 0 pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:02.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:02 smithi129 bash[53756]: audit 2023-07-29T17:17:01.314263+0000 mon.a (mon.0) 560 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:02.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:02 smithi129 bash[53756]: audit 2023-07-29T17:17:01.316496+0000 mon.a (mon.0) 561 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T17:17:02.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:02 smithi129 bash[53756]: audit 2023-07-29T17:17:01.318288+0000 mon.a (mon.0) 562 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:03.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:03 smithi129 bash[53756]: cephadm 2023-07-29T17:17:01.304304+0000 mgr.y (mgr.25169) 258 : cephadm 1 Upgrade: Updating osd.5 2023-07-29T17:17:03.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:03 smithi129 bash[53756]: cephadm 2023-07-29T17:17:01.324587+0000 mgr.y (mgr.25169) 259 : cephadm 1 Deploying daemon osd.5 on smithi129 2023-07-29T17:17:03.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:03 smithi078 bash[58614]: cephadm 2023-07-29T17:17:01.304304+0000 mgr.y (mgr.25169) 258 : cephadm 1 Upgrade: Updating osd.5 2023-07-29T17:17:03.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:03 smithi078 bash[58614]: cephadm 2023-07-29T17:17:01.324587+0000 mgr.y (mgr.25169) 259 : cephadm 1 Deploying daemon osd.5 on smithi129 2023-07-29T17:17:03.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:03 smithi078 bash[60360]: cephadm 2023-07-29T17:17:01.304304+0000 mgr.y (mgr.25169) 258 : cephadm 1 Upgrade: Updating osd.5 2023-07-29T17:17:03.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:03 smithi078 bash[60360]: cephadm 2023-07-29T17:17:01.324587+0000 mgr.y (mgr.25169) 259 : cephadm 1 Deploying daemon osd.5 on smithi129 2023-07-29T17:17:04.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:04 smithi129 bash[53756]: cluster 2023-07-29T17:17:02.972991+0000 mgr.y (mgr.25169) 260 : cluster 0 pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:04.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:04 smithi129 bash[53756]: audit 2023-07-29T17:17:03.460256+0000 mon.a (mon.0) 563 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:04.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:04 smithi078 bash[58614]: cluster 2023-07-29T17:17:02.972991+0000 mgr.y (mgr.25169) 260 : cluster 0 pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:04.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:04 smithi078 bash[58614]: audit 2023-07-29T17:17:03.460256+0000 mon.a (mon.0) 563 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:04.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:04 smithi078 bash[60360]: cluster 2023-07-29T17:17:02.972991+0000 mgr.y (mgr.25169) 260 : cluster 0 pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:04.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:04 smithi078 bash[60360]: audit 2023-07-29T17:17:03.460256+0000 mon.a (mon.0) 563 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:06.067 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:05 smithi129 systemd[1]: Stopping Ceph osd.5 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:17:06.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:06 smithi129 bash[53756]: cluster 2023-07-29T17:17:04.974519+0000 mgr.y (mgr.25169) 261 : cluster 0 pgmap v174: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:06.425 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:06 smithi129 bash[29407]: debug 2023-07-29T17:17:06.081+0000 7f2dde5ad700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:17:06.425 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:06 smithi129 bash[29407]: debug 2023-07-29T17:17:06.081+0000 7f2dde5ad700 -1 osd.5 103 *** Got signal Terminated *** 2023-07-29T17:17:06.425 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:06 smithi129 bash[29407]: debug 2023-07-29T17:17:06.081+0000 7f2dde5ad700 -1 osd.5 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:17:06.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:06 smithi078 bash[58614]: cluster 2023-07-29T17:17:04.974519+0000 mgr.y (mgr.25169) 261 : cluster 0 pgmap v174: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:06.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:06 smithi078 bash[60360]: cluster 2023-07-29T17:17:04.974519+0000 mgr.y (mgr.25169) 261 : cluster 0 pgmap v174: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:07.174 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:06 smithi129 bash[58100]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.5 2023-07-29T17:17:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.348442+0000 mon.a (mon.0) 564 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.348549+0000 mon.a (mon.0) 565 : cluster 1 osd.5 failed (root=default,host=smithi129) (connection refused reported by osd.1) 2023-07-29T17:17:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.349215+0000 mon.a (mon.0) 566 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.349631+0000 mon.a (mon.0) 567 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.350119+0000 mon.a (mon.0) 568 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.350319+0000 mon.a (mon.0) 569 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.350605+0000 mon.a (mon.0) 570 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.351014+0000 mon.a (mon.0) 571 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.351413+0000 mon.a (mon.0) 572 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.352083+0000 mon.a (mon.0) 573 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.352358+0000 mon.a (mon.0) 574 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.352757+0000 mon.a (mon.0) 575 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.353002+0000 mon.a (mon.0) 576 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.353422+0000 mon.a (mon.0) 577 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.353729+0000 mon.a (mon.0) 578 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.752394+0000 mon.a (mon.0) 579 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.752702+0000 mon.a (mon.0) 580 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.753259+0000 mon.a (mon.0) 581 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.753657+0000 mon.a (mon.0) 582 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.754161+0000 mon.a (mon.0) 583 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.754512+0000 mon.a (mon.0) 584 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.754863+0000 mon.a (mon.0) 585 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.755229+0000 mon.a (mon.0) 586 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.755845+0000 mon.a (mon.0) 587 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.756074+0000 mon.a (mon.0) 588 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.756676+0000 mon.a (mon.0) 589 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.757159+0000 mon.a (mon.0) 590 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.757373+0000 mon.a (mon.0) 591 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:07 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.757963+0000 mon.a (mon.0) 592 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.348442+0000 mon.a (mon.0) 564 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.348549+0000 mon.a (mon.0) 565 : cluster 1 osd.5 failed (root=default,host=smithi129) (connection refused reported by osd.1) 2023-07-29T17:17:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.349215+0000 mon.a (mon.0) 566 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.349631+0000 mon.a (mon.0) 567 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.350119+0000 mon.a (mon.0) 568 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.350319+0000 mon.a (mon.0) 569 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.350605+0000 mon.a (mon.0) 570 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.351014+0000 mon.a (mon.0) 571 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.351413+0000 mon.a (mon.0) 572 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.352083+0000 mon.a (mon.0) 573 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.352358+0000 mon.a (mon.0) 574 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.352757+0000 mon.a (mon.0) 575 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.353002+0000 mon.a (mon.0) 576 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.353422+0000 mon.a (mon.0) 577 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.353729+0000 mon.a (mon.0) 578 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.752394+0000 mon.a (mon.0) 579 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.752702+0000 mon.a (mon.0) 580 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.753259+0000 mon.a (mon.0) 581 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.753657+0000 mon.a (mon.0) 582 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.754161+0000 mon.a (mon.0) 583 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.754512+0000 mon.a (mon.0) 584 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.754863+0000 mon.a (mon.0) 585 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.755229+0000 mon.a (mon.0) 586 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.755845+0000 mon.a (mon.0) 587 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.756074+0000 mon.a (mon.0) 588 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.756676+0000 mon.a (mon.0) 589 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.757159+0000 mon.a (mon.0) 590 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.757373+0000 mon.a (mon.0) 591 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.757963+0000 mon.a (mon.0) 592 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.348442+0000 mon.a (mon.0) 564 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.348549+0000 mon.a (mon.0) 565 : cluster 1 osd.5 failed (root=default,host=smithi129) (connection refused reported by osd.1) 2023-07-29T17:17:07.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.349215+0000 mon.a (mon.0) 566 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.349631+0000 mon.a (mon.0) 567 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.350119+0000 mon.a (mon.0) 568 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.350319+0000 mon.a (mon.0) 569 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.350605+0000 mon.a (mon.0) 570 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.351014+0000 mon.a (mon.0) 571 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.351413+0000 mon.a (mon.0) 572 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.352083+0000 mon.a (mon.0) 573 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.352358+0000 mon.a (mon.0) 574 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.352757+0000 mon.a (mon.0) 575 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.353002+0000 mon.a (mon.0) 576 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.353422+0000 mon.a (mon.0) 577 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.353729+0000 mon.a (mon.0) 578 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.752394+0000 mon.a (mon.0) 579 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.752702+0000 mon.a (mon.0) 580 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.753259+0000 mon.a (mon.0) 581 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.753657+0000 mon.a (mon.0) 582 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.754161+0000 mon.a (mon.0) 583 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.754512+0000 mon.a (mon.0) 584 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.754863+0000 mon.a (mon.0) 585 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.755229+0000 mon.a (mon.0) 586 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.755845+0000 mon.a (mon.0) 587 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-29T17:17:07.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.756074+0000 mon.a (mon.0) 588 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.756676+0000 mon.a (mon.0) 589 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-29T17:17:07.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.757159+0000 mon.a (mon.0) 590 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:07.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.757373+0000 mon.a (mon.0) 591 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-29T17:17:07.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:07 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.757963+0000 mon.a (mon.0) 592 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-29T17:17:08.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:06.975255+0000 mgr.y (mgr.25169) 262 : cluster 0 pgmap v175: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:08.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.046754+0000 mon.a (mon.0) 593 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.063584+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.065189+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.067262+0000 mon.a (mon.0) 594 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.068014+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.068966+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.073941+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.074212+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.076202+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:08 smithi129 bash[53756]: cluster 2023-07-29T17:17:07.077134+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:06.975255+0000 mgr.y (mgr.25169) 262 : cluster 0 pgmap v175: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:08.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.046754+0000 mon.a (mon.0) 593 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:17:08.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.063584+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.065189+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.067262+0000 mon.a (mon.0) 594 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.068014+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.068966+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.073941+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.074212+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.076202+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[58614]: cluster 2023-07-29T17:17:07.077134+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:06.975255+0000 mgr.y (mgr.25169) 262 : cluster 0 pgmap v175: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:08.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.046754+0000 mon.a (mon.0) 593 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:17:08.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.063584+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.065189+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.067262+0000 mon.a (mon.0) 594 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-07-29T17:17:08.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.068014+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.068966+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.073941+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.074212+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.076202+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:08.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:08 smithi078 bash[60360]: cluster 2023-07-29T17:17:07.077134+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e104 with expected crc 2023-07-29T17:17:09.424 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:09 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.5.service: Succeeded. 2023-07-29T17:17:09.425 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:09 smithi129 systemd[1]: Stopped Ceph osd.5 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:17:09.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.070867+0000 mon.a (mon.0) 595 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-07-29T17:17:09.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.070931+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.071585+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.075776+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.078271+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.080325+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.081392+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.082483+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:09 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.084798+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.070867+0000 mon.a (mon.0) 595 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-07-29T17:17:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.070931+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.071585+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.075776+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.078271+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.080325+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.081392+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.082483+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.084798+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.070867+0000 mon.a (mon.0) 595 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-07-29T17:17:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.070931+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.071585+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.075776+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.078271+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.080325+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.081392+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.082483+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:09 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.084798+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e105 with expected crc 2023-07-29T17:17:09.924 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:09 smithi129 systemd[1]: Started Ceph osd.5 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:17:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:10 smithi129 bash[53756]: cluster 2023-07-29T17:17:08.976397+0000 mgr.y (mgr.25169) 263 : cluster 0 pgmap v178: 161 pgs: 5 active+undersized+degraded, 14 peering, 11 stale+active+clean, 5 active+undersized, 126 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 31/999 objects degraded (3.103%) 2023-07-29T17:17:10.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:10 smithi129 bash[53756]: audit 2023-07-29T17:17:09.583054+0000 mon.a (mon.0) 596 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:10.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:10 smithi129 bash[53756]: audit 2023-07-29T17:17:09.591739+0000 mon.a (mon.0) 597 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:10.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:10 smithi129 bash[53756]: cluster 2023-07-29T17:17:09.724691+0000 mon.a (mon.0) 598 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2023-07-29T17:17:10.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:10 smithi129 bash[53756]: cluster 2023-07-29T17:17:09.724766+0000 mon.a (mon.0) 599 : cluster 3 Health check failed: Degraded data redundancy: 31/999 objects degraded (3.103%), 5 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:10.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[60360]: cluster 2023-07-29T17:17:08.976397+0000 mgr.y (mgr.25169) 263 : cluster 0 pgmap v178: 161 pgs: 5 active+undersized+degraded, 14 peering, 11 stale+active+clean, 5 active+undersized, 126 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 31/999 objects degraded (3.103%) 2023-07-29T17:17:10.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[60360]: audit 2023-07-29T17:17:09.583054+0000 mon.a (mon.0) 596 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:10.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[60360]: audit 2023-07-29T17:17:09.591739+0000 mon.a (mon.0) 597 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:10.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[60360]: cluster 2023-07-29T17:17:09.724691+0000 mon.a (mon.0) 598 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2023-07-29T17:17:10.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[60360]: cluster 2023-07-29T17:17:09.724766+0000 mon.a (mon.0) 599 : cluster 3 Health check failed: Degraded data redundancy: 31/999 objects degraded (3.103%), 5 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:10.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[58614]: cluster 2023-07-29T17:17:08.976397+0000 mgr.y (mgr.25169) 263 : cluster 0 pgmap v178: 161 pgs: 5 active+undersized+degraded, 14 peering, 11 stale+active+clean, 5 active+undersized, 126 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 31/999 objects degraded (3.103%) 2023-07-29T17:17:10.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[58614]: audit 2023-07-29T17:17:09.583054+0000 mon.a (mon.0) 596 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:10.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[58614]: audit 2023-07-29T17:17:09.591739+0000 mon.a (mon.0) 597 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:10.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[58614]: cluster 2023-07-29T17:17:09.724691+0000 mon.a (mon.0) 598 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2023-07-29T17:17:10.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:10 smithi078 bash[58614]: cluster 2023-07-29T17:17:09.724766+0000 mon.a (mon.0) 599 : cluster 3 Health check failed: Degraded data redundancy: 31/999 objects degraded (3.103%), 5 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:11.674 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-29T17:17:11.675 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-07-29T17:17:11.675 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-07-29T17:17:11.675 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-29T17:17:11.675 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2023-07-29T17:17:11.675 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-29T17:17:11.675 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:11 smithi129 bash[58370]: --> ceph-volume raw activate successful for osd ID: 5 2023-07-29T17:17:12.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:12 smithi129 bash[53756]: cluster 2023-07-29T17:17:10.977993+0000 mgr.y (mgr.25169) 264 : cluster 0 pgmap v179: 161 pgs: 15 active+undersized+degraded, 14 peering, 30 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 98/999 objects degraded (9.810%) 2023-07-29T17:17:12.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:12 smithi078 bash[58614]: cluster 2023-07-29T17:17:10.977993+0000 mgr.y (mgr.25169) 264 : cluster 0 pgmap v179: 161 pgs: 15 active+undersized+degraded, 14 peering, 30 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 98/999 objects degraded (9.810%) 2023-07-29T17:17:12.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:12 smithi078 bash[60360]: cluster 2023-07-29T17:17:10.977993+0000 mgr.y (mgr.25169) 264 : cluster 0 pgmap v179: 161 pgs: 15 active+undersized+degraded, 14 peering, 30 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 98/999 objects degraded (9.810%) 2023-07-29T17:17:13.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:13 smithi129 bash[53756]: cluster 2023-07-29T17:17:13.076105+0000 mon.a (mon.0) 600 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2023-07-29T17:17:13.425 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:13 smithi129 bash[58650]: debug 2023-07-29T17:17:13.209+0000 7f7b2552b700 1 -- 172.21.15.129:0/2895130845 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56427c2fa4e0 con 0x56427c354800 2023-07-29T17:17:13.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:13 smithi078 bash[58614]: cluster 2023-07-29T17:17:13.076105+0000 mon.a (mon.0) 600 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2023-07-29T17:17:13.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:13 smithi078 bash[60360]: cluster 2023-07-29T17:17:13.076105+0000 mon.a (mon.0) 600 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2023-07-29T17:17:14.154 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:13 smithi129 bash[58650]: debug 2023-07-29T17:17:13.753+0000 7f7b2e6b0640 -1 Falling back to public interface 2023-07-29T17:17:14.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:14 smithi129 bash[53756]: cluster 2023-07-29T17:17:12.979266+0000 mgr.y (mgr.25169) 265 : cluster 0 pgmap v180: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:14.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:14 smithi078 bash[58614]: cluster 2023-07-29T17:17:12.979266+0000 mgr.y (mgr.25169) 265 : cluster 0 pgmap v180: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:14.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:14 smithi078 bash[60360]: cluster 2023-07-29T17:17:12.979266+0000 mgr.y (mgr.25169) 265 : cluster 0 pgmap v180: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:16.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:16 smithi129 bash[53756]: cluster 2023-07-29T17:17:14.980986+0000 mgr.y (mgr.25169) 266 : cluster 0 pgmap v181: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:16.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:16 smithi078 bash[58614]: cluster 2023-07-29T17:17:14.980986+0000 mgr.y (mgr.25169) 266 : cluster 0 pgmap v181: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:16.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:16 smithi078 bash[60360]: cluster 2023-07-29T17:17:14.980986+0000 mgr.y (mgr.25169) 266 : cluster 0 pgmap v181: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:17.168 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:16 smithi129 bash[58650]: debug 2023-07-29T17:17:16.797+0000 7f7b2e6b0640 -1 osd.5 103 log_to_monitors true 2023-07-29T17:17:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:17 smithi129 bash[53756]: audit 2023-07-29T17:17:16.804912+0000 mon.a (mon.0) 601 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:17:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:17 smithi129 bash[53756]: audit 2023-07-29T17:17:16.806541+0000 mon.b (mon.1) 5 : audit 1 from='osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:17:17.424 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:17:17 smithi129 bash[58650]: debug 2023-07-29T17:17:17.197+0000 7f7b1d318700 -1 osd.5 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:17:17.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:17 smithi078 bash[60360]: audit 2023-07-29T17:17:16.804912+0000 mon.a (mon.0) 601 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:17:17.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:17 smithi078 bash[60360]: audit 2023-07-29T17:17:16.806541+0000 mon.b (mon.1) 5 : audit 1 from='osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:17:17.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:17 smithi078 bash[58614]: audit 2023-07-29T17:17:16.804912+0000 mon.a (mon.0) 601 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:17:17.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:17 smithi078 bash[58614]: audit 2023-07-29T17:17:16.806541+0000 mon.b (mon.1) 5 : audit 1 from='osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T17:17:18.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: cluster 2023-07-29T17:17:16.981947+0000 mgr.y (mgr.25169) 267 : cluster 0 pgmap v182: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: cluster 2023-07-29T17:17:17.149958+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e106 with expected crc 2023-07-29T17:17:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:17.152215+0000 mon.a (mon.0) 602 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T17:17:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: cluster 2023-07-29T17:17:17.152316+0000 mon.a (mon.0) 603 : cluster 0 osdmap e106: 8 total, 7 up, 8 in 2023-07-29T17:17:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:17.153899+0000 mon.a (mon.0) 604 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:17.155553+0000 mon.b (mon.1) 6 : audit 1 from='osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:17.257495+0000 mon.a (mon.0) 605 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:17.267275+0000 mon.a (mon.0) 606 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:18.060747+0000 mon.a (mon.0) 607 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[58614]: audit 2023-07-29T17:17:18.072511+0000 mon.a (mon.0) 608 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: cluster 2023-07-29T17:17:16.981947+0000 mgr.y (mgr.25169) 267 : cluster 0 pgmap v182: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:18.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: cluster 2023-07-29T17:17:17.149958+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e106 with expected crc 2023-07-29T17:17:18.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:17.152215+0000 mon.a (mon.0) 602 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: cluster 2023-07-29T17:17:17.152316+0000 mon.a (mon.0) 603 : cluster 0 osdmap e106: 8 total, 7 up, 8 in 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:17.153899+0000 mon.a (mon.0) 604 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:17.155553+0000 mon.b (mon.1) 6 : audit 1 from='osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:17.257495+0000 mon.a (mon.0) 605 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:17.267275+0000 mon.a (mon.0) 606 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:18.060747+0000 mon.a (mon.0) 607 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:18 smithi078 bash[60360]: audit 2023-07-29T17:17:18.072511+0000 mon.a (mon.0) 608 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: cluster 2023-07-29T17:17:16.981947+0000 mgr.y (mgr.25169) 267 : cluster 0 pgmap v182: 161 pgs: 18 active+undersized+degraded, 41 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-29T17:17:18.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: cluster 2023-07-29T17:17:17.149958+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e106 with expected crc 2023-07-29T17:17:18.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:17.152215+0000 mon.a (mon.0) 602 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T17:17:18.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: cluster 2023-07-29T17:17:17.152316+0000 mon.a (mon.0) 603 : cluster 0 osdmap e106: 8 total, 7 up, 8 in 2023-07-29T17:17:18.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:17.153899+0000 mon.a (mon.0) 604 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:18.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:17.155553+0000 mon.b (mon.1) 6 : audit 1 from='osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:18.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:17.257495+0000 mon.a (mon.0) 605 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:17.267275+0000 mon.a (mon.0) 606 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:18.060747+0000 mon.a (mon.0) 607 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:18.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:18 smithi129 bash[53756]: audit 2023-07-29T17:17:18.072511+0000 mon.a (mon.0) 608 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:19.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:17.775771+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e106 with expected crc 2023-07-29T17:17:19.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.145544+0000 mon.a (mon.0) 609 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:17:19.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.152911+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.156459+0000 mon.a (mon.0) 610 : cluster 1 osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664] boot 2023-07-29T17:17:19.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.157073+0000 mon.a (mon.0) 611 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-07-29T17:17:19.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.157868+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.160482+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.161164+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: audit 2023-07-29T17:17:18.161330+0000 mon.a (mon.0) 612 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.163781+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.167577+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: audit 2023-07-29T17:17:18.489812+0000 mon.a (mon.0) 613 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:19.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.724807+0000 mon.a (mon.0) 614 : cluster 3 Health check update: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:19.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.156027+0000 mon.a (mon.0) 615 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2023-07-29T17:17:19.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:17.775771+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e106 with expected crc 2023-07-29T17:17:19.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.145544+0000 mon.a (mon.0) 609 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:17:19.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.152911+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.156459+0000 mon.a (mon.0) 610 : cluster 1 osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664] boot 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.157073+0000 mon.a (mon.0) 611 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.157868+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.160482+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.161164+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: audit 2023-07-29T17:17:18.161330+0000 mon.a (mon.0) 612 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.163781+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.167577+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: audit 2023-07-29T17:17:18.489812+0000 mon.a (mon.0) 613 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:19.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.724807+0000 mon.a (mon.0) 614 : cluster 3 Health check update: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:19.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:19 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.156027+0000 mon.a (mon.0) 615 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2023-07-29T17:17:19.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:17.775771+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e106 with expected crc 2023-07-29T17:17:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.145544+0000 mon.a (mon.0) 609 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:17:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.152911+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.156459+0000 mon.a (mon.0) 610 : cluster 1 osd.5 [v2:172.21.15.129:6808/3958496664,v1:172.21.15.129:6809/3958496664] boot 2023-07-29T17:17:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.157073+0000 mon.a (mon.0) 611 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-07-29T17:17:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.157868+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.160482+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.161164+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: audit 2023-07-29T17:17:18.161330+0000 mon.a (mon.0) 612 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.163781+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.167577+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e107 with expected crc 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: audit 2023-07-29T17:17:18.489812+0000 mon.a (mon.0) 613 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.724807+0000 mon.a (mon.0) 614 : cluster 3 Health check update: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:19.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:19 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.156027+0000 mon.a (mon.0) 615 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2023-07-29T17:17:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:18.983190+0000 mgr.y (mgr.25169) 268 : cluster 0 pgmap v185: 161 pgs: 5 peering, 16 active+undersized+degraded, 38 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 104/999 objects degraded (10.410%) 2023-07-29T17:17:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.154102+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.154744+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.159003+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.160094+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.166589+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.168519+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.170414+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:20 smithi129 bash[53756]: cluster 2023-07-29T17:17:19.172416+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:18.983190+0000 mgr.y (mgr.25169) 268 : cluster 0 pgmap v185: 161 pgs: 5 peering, 16 active+undersized+degraded, 38 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 104/999 objects degraded (10.410%) 2023-07-29T17:17:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.154102+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.154744+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.159003+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.160094+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.166589+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.168519+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.170414+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[58614]: cluster 2023-07-29T17:17:19.172416+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:18.983190+0000 mgr.y (mgr.25169) 268 : cluster 0 pgmap v185: 161 pgs: 5 peering, 16 active+undersized+degraded, 38 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 104/999 objects degraded (10.410%) 2023-07-29T17:17:20.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.154102+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.154744+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.159003+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.160094+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.166589+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.168519+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.170414+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:20.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:20 smithi078 bash[60360]: cluster 2023-07-29T17:17:19.172416+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e108 with expected crc 2023-07-29T17:17:22.674 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:22 smithi129 bash[53756]: cluster 2023-07-29T17:17:20.984702+0000 mgr.y (mgr.25169) 269 : cluster 0 pgmap v187: 161 pgs: 6 activating, 5 peering, 8 active+undersized+degraded, 22 active+undersized, 120 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:17:22.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:22 smithi078 bash[58614]: cluster 2023-07-29T17:17:20.984702+0000 mgr.y (mgr.25169) 269 : cluster 0 pgmap v187: 161 pgs: 6 activating, 5 peering, 8 active+undersized+degraded, 22 active+undersized, 120 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:17:22.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:22 smithi078 bash[60360]: cluster 2023-07-29T17:17:20.984702+0000 mgr.y (mgr.25169) 269 : cluster 0 pgmap v187: 161 pgs: 6 activating, 5 peering, 8 active+undersized+degraded, 22 active+undersized, 120 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 49/999 objects degraded (4.905%) 2023-07-29T17:17:23.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:23 smithi129 bash[53756]: cluster 2023-07-29T17:17:22.985830+0000 mgr.y (mgr.25169) 270 : cluster 0 pgmap v188: 161 pgs: 6 activating, 5 peering, 150 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:23.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:23 smithi129 bash[53756]: cluster 2023-07-29T17:17:23.176396+0000 mon.a (mon.0) 616 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/999 objects degraded (4.905%), 8 pgs degraded) 2023-07-29T17:17:23.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:23 smithi078 bash[58614]: cluster 2023-07-29T17:17:22.985830+0000 mgr.y (mgr.25169) 270 : cluster 0 pgmap v188: 161 pgs: 6 activating, 5 peering, 150 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:23.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:23 smithi078 bash[58614]: cluster 2023-07-29T17:17:23.176396+0000 mon.a (mon.0) 616 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/999 objects degraded (4.905%), 8 pgs degraded) 2023-07-29T17:17:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:23 smithi078 bash[60360]: cluster 2023-07-29T17:17:22.985830+0000 mgr.y (mgr.25169) 270 : cluster 0 pgmap v188: 161 pgs: 6 activating, 5 peering, 150 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:23 smithi078 bash[60360]: cluster 2023-07-29T17:17:23.176396+0000 mon.a (mon.0) 616 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/999 objects degraded (4.905%), 8 pgs degraded) 2023-07-29T17:17:26.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:26 smithi129 bash[53756]: cluster 2023-07-29T17:17:24.987251+0000 mgr.y (mgr.25169) 271 : cluster 0 pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:26.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:26 smithi078 bash[58614]: cluster 2023-07-29T17:17:24.987251+0000 mgr.y (mgr.25169) 271 : cluster 0 pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:26.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:26 smithi078 bash[60360]: cluster 2023-07-29T17:17:24.987251+0000 mgr.y (mgr.25169) 271 : cluster 0 pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:27.347 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:17:28.083 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:17:28.083 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7m) 70s ago 13m 10.7M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:17:28.083 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (6m) 10s ago 13m 26.3M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:17:28.083 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 10s ago 11m - - 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (5m) 10s ago 17m 429M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (6m) 70s ago 19m 473M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5m) 70s ago 19m 64.3M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (3m) 10s ago 18m 52.7M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4m) 70s ago 18m 48.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7m) 70s ago 13m 8844k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:17:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (7m) 10s ago 13m 3760k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2m) 70s ago 17m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2m) 70s ago 17m 52.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (105s) 70s ago 16m 46.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (75s) 70s ago 16m - 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (45s) 10s ago 15m 51.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (15s) 10s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (15m) 10s ago 15m 61.7M 4096M 16.2.0 24ecd6d5f14c 1e92339f2c93 2023-07-29T17:17:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (14m) 10s ago 14m 65.3M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:17:28.086 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 10s ago 13m - - 2023-07-29T17:17:28.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (11m) 70s ago 11m 55.8M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:17:28.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (11m) 10s ago 11m 56.0M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:17:28.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (11m) 70s ago 11m 56.0M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:17:28.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (11m) 10s ago 11m 56.0M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:17:28.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:28 smithi129 bash[53756]: cluster 2023-07-29T17:17:26.988039+0000 mgr.y (mgr.25169) 272 : cluster 0 pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:28.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:28 smithi078 bash[60360]: cluster 2023-07-29T17:17:26.988039+0000 mgr.y (mgr.25169) 272 : cluster 0 pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:28.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:28 smithi078 bash[58614]: cluster 2023-07-29T17:17:26.988039+0000 mgr.y (mgr.25169) 272 : cluster 0 pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:28.532 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:17:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:17:28.532 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2023-07-29T17:17:28.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 6 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 11 2023-07-29T17:17:28.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:17:28.535 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:17:28.908 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:17:28.908 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:17:28.908 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:17:28.908 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:17:28.908 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "11/23 daemons upgraded", 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:17:28.909 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:17:29.338 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:17:29.338 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:17:29.338 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:17:29.338 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:17:29.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:27.310024+0000 mgr.y (mgr.25169) 273 : audit 0 from='client.44390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:27.681608+0000 mgr.y (mgr.25169) 274 : audit 0 from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.038733+0000 mgr.y (mgr.25169) 275 : audit 0 from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.275231+0000 mon.a (mon.0) 617 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.287441+0000 mon.a (mon.0) 618 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.289052+0000 mon.a (mon.0) 619 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.289966+0000 mon.a (mon.0) 620 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.299992+0000 mon.a (mon.0) 621 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.380712+0000 mon.a (mon.0) 622 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:29.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.384839+0000 mon.a (mon.0) 623 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.387108+0000 mon.a (mon.0) 624 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.388965+0000 mon.a (mon.0) 625 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.391173+0000 mon.a (mon.0) 626 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-29T17:17:29.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.509656+0000 mon.c (mon.2) 10 : audit 0 from='client.? 172.21.15.78:0/171041390' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:29 smithi129 bash[53756]: audit 2023-07-29T17:17:28.509672+0000 mon.a (mon.0) 627 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:27.310024+0000 mgr.y (mgr.25169) 273 : audit 0 from='client.44390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:27.681608+0000 mgr.y (mgr.25169) 274 : audit 0 from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.038733+0000 mgr.y (mgr.25169) 275 : audit 0 from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.275231+0000 mon.a (mon.0) 617 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.287441+0000 mon.a (mon.0) 618 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.289052+0000 mon.a (mon.0) 619 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.289966+0000 mon.a (mon.0) 620 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.299992+0000 mon.a (mon.0) 621 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.380712+0000 mon.a (mon.0) 622 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.384839+0000 mon.a (mon.0) 623 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.387108+0000 mon.a (mon.0) 624 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.388965+0000 mon.a (mon.0) 625 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.391173+0000 mon.a (mon.0) 626 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-29T17:17:29.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.509656+0000 mon.c (mon.2) 10 : audit 0 from='client.? 172.21.15.78:0/171041390' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[58614]: audit 2023-07-29T17:17:28.509672+0000 mon.a (mon.0) 627 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:27.310024+0000 mgr.y (mgr.25169) 273 : audit 0 from='client.44390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:27.681608+0000 mgr.y (mgr.25169) 274 : audit 0 from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.038733+0000 mgr.y (mgr.25169) 275 : audit 0 from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:29.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.275231+0000 mon.a (mon.0) 617 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.287441+0000 mon.a (mon.0) 618 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.289052+0000 mon.a (mon.0) 619 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.289966+0000 mon.a (mon.0) 620 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.299992+0000 mon.a (mon.0) 621 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.380712+0000 mon.a (mon.0) 622 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.384839+0000 mon.a (mon.0) 623 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.387108+0000 mon.a (mon.0) 624 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.388965+0000 mon.a (mon.0) 625 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.391173+0000 mon.a (mon.0) 626 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-29T17:17:29.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.509656+0000 mon.c (mon.2) 10 : audit 0 from='client.? 172.21.15.78:0/171041390' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:29.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:29 smithi078 bash[60360]: audit 2023-07-29T17:17:28.509672+0000 mon.a (mon.0) 627 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:30.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: cephadm 2023-07-29T17:17:28.290797+0000 mgr.y (mgr.25169) 276 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: cluster 2023-07-29T17:17:28.291788+0000 mgr.y (mgr.25169) 277 : cluster 0 pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: cluster 2023-07-29T17:17:28.292075+0000 mgr.y (mgr.25169) 278 : cluster 0 pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: audit 2023-07-29T17:17:28.391547+0000 mgr.y (mgr.25169) 279 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-29T17:17:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: cephadm 2023-07-29T17:17:28.393094+0000 mgr.y (mgr.25169) 280 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-07-29T17:17:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: audit 2023-07-29T17:17:28.885106+0000 mgr.y (mgr.25169) 281 : audit 0 from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:30 smithi129 bash[53756]: audit 2023-07-29T17:17:29.315787+0000 mon.a (mon.0) 628 : audit 0 from='client.34590 172.21.15.78:0/953792207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:17:30.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: cephadm 2023-07-29T17:17:28.290797+0000 mgr.y (mgr.25169) 276 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:30.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: cluster 2023-07-29T17:17:28.291788+0000 mgr.y (mgr.25169) 277 : cluster 0 pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:30.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: cluster 2023-07-29T17:17:28.292075+0000 mgr.y (mgr.25169) 278 : cluster 0 pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:30.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: audit 2023-07-29T17:17:28.391547+0000 mgr.y (mgr.25169) 279 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-29T17:17:30.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: cephadm 2023-07-29T17:17:28.393094+0000 mgr.y (mgr.25169) 280 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-07-29T17:17:30.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: audit 2023-07-29T17:17:28.885106+0000 mgr.y (mgr.25169) 281 : audit 0 from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:30.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[58614]: audit 2023-07-29T17:17:29.315787+0000 mon.a (mon.0) 628 : audit 0 from='client.34590 172.21.15.78:0/953792207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:17:30.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: cephadm 2023-07-29T17:17:28.290797+0000 mgr.y (mgr.25169) 276 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:30.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: cluster 2023-07-29T17:17:28.291788+0000 mgr.y (mgr.25169) 277 : cluster 0 pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:30.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: cluster 2023-07-29T17:17:28.292075+0000 mgr.y (mgr.25169) 278 : cluster 0 pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:30.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: audit 2023-07-29T17:17:28.391547+0000 mgr.y (mgr.25169) 279 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-29T17:17:30.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: cephadm 2023-07-29T17:17:28.393094+0000 mgr.y (mgr.25169) 280 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-07-29T17:17:30.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: audit 2023-07-29T17:17:28.885106+0000 mgr.y (mgr.25169) 281 : audit 0 from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:17:30.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:30 smithi078 bash[60360]: audit 2023-07-29T17:17:29.315787+0000 mon.a (mon.0) 628 : audit 0 from='client.34590 172.21.15.78:0/953792207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:17:32.097 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:31 smithi129 bash[53756]: cluster 2023-07-29T17:17:30.293322+0000 mgr.y (mgr.25169) 282 : cluster 0 pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:32.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:31 smithi129 bash[53756]: cephadm 2023-07-29T17:17:30.812675+0000 mgr.y (mgr.25169) 283 : cephadm 1 Upgrade: Updating osd.6 2023-07-29T17:17:32.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:31 smithi129 bash[53756]: audit 2023-07-29T17:17:30.822536+0000 mon.a (mon.0) 629 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:32.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:31 smithi129 bash[53756]: audit 2023-07-29T17:17:30.824746+0000 mon.a (mon.0) 630 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:17:32.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:31 smithi129 bash[53756]: audit 2023-07-29T17:17:30.826427+0000 mon.a (mon.0) 631 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:32.098 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:31 smithi129 bash[53756]: cephadm 2023-07-29T17:17:30.832136+0000 mgr.y (mgr.25169) 284 : cephadm 1 Deploying daemon osd.6 on smithi129 2023-07-29T17:17:32.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[58614]: cluster 2023-07-29T17:17:30.293322+0000 mgr.y (mgr.25169) 282 : cluster 0 pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:32.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[58614]: cephadm 2023-07-29T17:17:30.812675+0000 mgr.y (mgr.25169) 283 : cephadm 1 Upgrade: Updating osd.6 2023-07-29T17:17:32.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[58614]: audit 2023-07-29T17:17:30.822536+0000 mon.a (mon.0) 629 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:32.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[58614]: audit 2023-07-29T17:17:30.824746+0000 mon.a (mon.0) 630 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:17:32.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[58614]: audit 2023-07-29T17:17:30.826427+0000 mon.a (mon.0) 631 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:32.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[58614]: cephadm 2023-07-29T17:17:30.832136+0000 mgr.y (mgr.25169) 284 : cephadm 1 Deploying daemon osd.6 on smithi129 2023-07-29T17:17:32.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[60360]: cluster 2023-07-29T17:17:30.293322+0000 mgr.y (mgr.25169) 282 : cluster 0 pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:32.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[60360]: cephadm 2023-07-29T17:17:30.812675+0000 mgr.y (mgr.25169) 283 : cephadm 1 Upgrade: Updating osd.6 2023-07-29T17:17:32.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[60360]: audit 2023-07-29T17:17:30.822536+0000 mon.a (mon.0) 629 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:32.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[60360]: audit 2023-07-29T17:17:30.824746+0000 mon.a (mon.0) 630 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T17:17:32.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[60360]: audit 2023-07-29T17:17:30.826427+0000 mon.a (mon.0) 631 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:32.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:31 smithi078 bash[60360]: cephadm 2023-07-29T17:17:30.832136+0000 mgr.y (mgr.25169) 284 : cephadm 1 Deploying daemon osd.6 on smithi129 2023-07-29T17:17:34.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:33 smithi129 bash[53756]: cluster 2023-07-29T17:17:32.294147+0000 mgr.y (mgr.25169) 285 : cluster 0 pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:34.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:33 smithi078 bash[58614]: cluster 2023-07-29T17:17:32.294147+0000 mgr.y (mgr.25169) 285 : cluster 0 pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:34.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:33 smithi078 bash[60360]: cluster 2023-07-29T17:17:32.294147+0000 mgr.y (mgr.25169) 285 : cluster 0 pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:35.424 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:35 smithi129 systemd[1]: Stopping Ceph osd.6 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:17:35.851 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[32184]: debug 2023-07-29T17:17:35.444+0000 7f77aa741700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:17:35.851 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[32184]: debug 2023-07-29T17:17:35.444+0000 7f77aa741700 -1 osd.6 108 *** Got signal Terminated *** 2023-07-29T17:17:35.851 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[32184]: debug 2023-07-29T17:17:35.444+0000 7f77aa741700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:17:36.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:34.295751+0000 mgr.y (mgr.25169) 286 : cluster 0 pgmap v195: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:36.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.704384+0000 mon.a (mon.0) 632 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-29T17:17:36.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.704488+0000 mon.a (mon.0) 633 : cluster 1 osd.6 failed (root=default,host=smithi129) (connection refused reported by osd.3) 2023-07-29T17:17:36.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.705012+0000 mon.a (mon.0) 634 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-29T17:17:36.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.705268+0000 mon.a (mon.0) 635 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-29T17:17:36.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.705482+0000 mon.a (mon.0) 636 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-29T17:17:36.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.705859+0000 mon.a (mon.0) 637 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-29T17:17:36.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.706000+0000 mon.a (mon.0) 638 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-29T17:17:36.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.706356+0000 mon.a (mon.0) 639 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-29T17:17:36.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.706660+0000 mon.a (mon.0) 640 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-29T17:17:36.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.706943+0000 mon.a (mon.0) 641 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-29T17:17:36.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.707175+0000 mon.a (mon.0) 642 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-29T17:17:36.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.707456+0000 mon.a (mon.0) 643 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-29T17:17:36.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.707716+0000 mon.a (mon.0) 644 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-29T17:17:36.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.707985+0000 mon.a (mon.0) 645 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-29T17:17:36.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:35 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.708278+0000 mon.a (mon.0) 646 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-29T17:17:36.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:34.295751+0000 mgr.y (mgr.25169) 286 : cluster 0 pgmap v195: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.704384+0000 mon.a (mon.0) 632 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-29T17:17:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.704488+0000 mon.a (mon.0) 633 : cluster 1 osd.6 failed (root=default,host=smithi129) (connection refused reported by osd.3) 2023-07-29T17:17:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.705012+0000 mon.a (mon.0) 634 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-29T17:17:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.705268+0000 mon.a (mon.0) 635 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-29T17:17:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.705482+0000 mon.a (mon.0) 636 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.705859+0000 mon.a (mon.0) 637 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.706000+0000 mon.a (mon.0) 638 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.706356+0000 mon.a (mon.0) 639 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.706660+0000 mon.a (mon.0) 640 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.706943+0000 mon.a (mon.0) 641 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.707175+0000 mon.a (mon.0) 642 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.707456+0000 mon.a (mon.0) 643 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-29T17:17:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.707716+0000 mon.a (mon.0) 644 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-29T17:17:36.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.707985+0000 mon.a (mon.0) 645 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-29T17:17:36.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.708278+0000 mon.a (mon.0) 646 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-29T17:17:36.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:34.295751+0000 mgr.y (mgr.25169) 286 : cluster 0 pgmap v195: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:36.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.704384+0000 mon.a (mon.0) 632 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-29T17:17:36.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.704488+0000 mon.a (mon.0) 633 : cluster 1 osd.6 failed (root=default,host=smithi129) (connection refused reported by osd.3) 2023-07-29T17:17:36.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.705012+0000 mon.a (mon.0) 634 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.705268+0000 mon.a (mon.0) 635 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.705482+0000 mon.a (mon.0) 636 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.705859+0000 mon.a (mon.0) 637 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.706000+0000 mon.a (mon.0) 638 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.706356+0000 mon.a (mon.0) 639 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.706660+0000 mon.a (mon.0) 640 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-29T17:17:36.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.706943+0000 mon.a (mon.0) 641 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-29T17:17:36.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.707175+0000 mon.a (mon.0) 642 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-29T17:17:36.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.707456+0000 mon.a (mon.0) 643 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-29T17:17:36.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.707716+0000 mon.a (mon.0) 644 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-29T17:17:36.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.707985+0000 mon.a (mon.0) 645 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-29T17:17:36.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:35 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.708278+0000 mon.a (mon.0) 646 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-29T17:17:36.674 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[60287]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.6 2023-07-29T17:17:36.675 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[60363]: Error response from daemon: No such container: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.6 2023-07-29T17:17:37.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.826901+0000 mon.a (mon.0) 647 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:17:37.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.842018+0000 mon.a (mon.0) 648 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-07-29T17:17:37.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.847590+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.850023+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:36 smithi129 bash[53756]: cluster 2023-07-29T17:17:35.852116+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.826901+0000 mon.a (mon.0) 647 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:17:37.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.842018+0000 mon.a (mon.0) 648 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-07-29T17:17:37.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.847590+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.850023+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[60360]: cluster 2023-07-29T17:17:35.852116+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.826901+0000 mon.a (mon.0) 647 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:17:37.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.842018+0000 mon.a (mon.0) 648 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-07-29T17:17:37.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.847590+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.850023+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:37.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:36 smithi078 bash[58614]: cluster 2023-07-29T17:17:35.852116+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e109 with expected crc 2023-07-29T17:17:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:37 smithi129 bash[53756]: cluster 2023-07-29T17:17:36.296543+0000 mgr.y (mgr.25169) 287 : cluster 0 pgmap v197: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:37 smithi129 bash[53756]: cluster 2023-07-29T17:17:36.852541+0000 mon.a (mon.0) 649 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-07-29T17:17:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:37 smithi129 bash[53756]: cluster 2023-07-29T17:17:36.855656+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e110 with expected crc 2023-07-29T17:17:38.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:37 smithi129 bash[53756]: cluster 2023-07-29T17:17:36.858340+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e110 with expected crc 2023-07-29T17:17:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[58614]: cluster 2023-07-29T17:17:36.296543+0000 mgr.y (mgr.25169) 287 : cluster 0 pgmap v197: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:38.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[58614]: cluster 2023-07-29T17:17:36.852541+0000 mon.a (mon.0) 649 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-07-29T17:17:38.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[58614]: cluster 2023-07-29T17:17:36.855656+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e110 with expected crc 2023-07-29T17:17:38.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[58614]: cluster 2023-07-29T17:17:36.858340+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e110 with expected crc 2023-07-29T17:17:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[60360]: cluster 2023-07-29T17:17:36.296543+0000 mgr.y (mgr.25169) 287 : cluster 0 pgmap v197: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[60360]: cluster 2023-07-29T17:17:36.852541+0000 mon.a (mon.0) 649 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-07-29T17:17:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[60360]: cluster 2023-07-29T17:17:36.855656+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e110 with expected crc 2023-07-29T17:17:38.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:37 smithi078 bash[60360]: cluster 2023-07-29T17:17:36.858340+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e110 with expected crc 2023-07-29T17:17:38.868 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:38 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.6.service: Succeeded. 2023-07-29T17:17:38.868 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:38 smithi129 systemd[1]: Stopped Ceph osd.6 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:17:39.174 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:38 smithi129 systemd[1]: Started Ceph osd.6 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:17:40.063 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:39 smithi129 bash[53756]: cluster 2023-07-29T17:17:38.298009+0000 mgr.y (mgr.25169) 288 : cluster 0 pgmap v199: 161 pgs: 11 active+undersized+degraded, 4 stale+active+clean, 24 active+undersized, 122 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2023-07-29T17:17:40.063 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:39 smithi129 bash[53756]: cluster 2023-07-29T17:17:38.844068+0000 mon.a (mon.0) 650 : cluster 3 Health check failed: Degraded data redundancy: 77/999 objects degraded (7.708%), 11 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:40.063 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:39 smithi129 bash[53756]: audit 2023-07-29T17:17:38.888435+0000 mon.a (mon.0) 651 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:40.063 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:39 smithi129 bash[53756]: audit 2023-07-29T17:17:38.896308+0000 mon.a (mon.0) 652 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:40.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[58614]: cluster 2023-07-29T17:17:38.298009+0000 mgr.y (mgr.25169) 288 : cluster 0 pgmap v199: 161 pgs: 11 active+undersized+degraded, 4 stale+active+clean, 24 active+undersized, 122 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2023-07-29T17:17:40.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[58614]: cluster 2023-07-29T17:17:38.844068+0000 mon.a (mon.0) 650 : cluster 3 Health check failed: Degraded data redundancy: 77/999 objects degraded (7.708%), 11 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:40.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[58614]: audit 2023-07-29T17:17:38.888435+0000 mon.a (mon.0) 651 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:40.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[58614]: audit 2023-07-29T17:17:38.896308+0000 mon.a (mon.0) 652 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:40.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[60360]: cluster 2023-07-29T17:17:38.298009+0000 mgr.y (mgr.25169) 288 : cluster 0 pgmap v199: 161 pgs: 11 active+undersized+degraded, 4 stale+active+clean, 24 active+undersized, 122 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2023-07-29T17:17:40.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[60360]: cluster 2023-07-29T17:17:38.844068+0000 mon.a (mon.0) 650 : cluster 3 Health check failed: Degraded data redundancy: 77/999 objects degraded (7.708%), 11 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:40.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[60360]: audit 2023-07-29T17:17:38.888435+0000 mon.a (mon.0) 651 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:40.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:39 smithi078 bash[60360]: audit 2023-07-29T17:17:38.896308+0000 mon.a (mon.0) 652 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:40.924 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-29T17:17:40.925 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-07-29T17:17:40.925 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-07-29T17:17:40.925 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-29T17:17:40.925 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2023-07-29T17:17:40.925 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-29T17:17:40.925 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:40 smithi129 bash[60561]: --> ceph-volume raw activate successful for osd ID: 6 2023-07-29T17:17:42.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:41 smithi129 bash[53756]: cluster 2023-07-29T17:17:40.299283+0000 mgr.y (mgr.25169) 289 : cluster 0 pgmap v200: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:42.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:41 smithi078 bash[58614]: cluster 2023-07-29T17:17:40.299283+0000 mgr.y (mgr.25169) 289 : cluster 0 pgmap v200: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:42.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:41 smithi078 bash[60360]: cluster 2023-07-29T17:17:40.299283+0000 mgr.y (mgr.25169) 289 : cluster 0 pgmap v200: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:42.674 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:42 smithi129 bash[60866]: debug 2023-07-29T17:17:42.412+0000 7f95aac77700 1 -- 172.21.15.129:0/656682173 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca99b844e0 con 0x55ca99bdf000 2023-07-29T17:17:43.424 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:42 smithi129 bash[60866]: debug 2023-07-29T17:17:42.944+0000 7f95b3dfc640 -1 Falling back to public interface 2023-07-29T17:17:44.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:43 smithi129 bash[53756]: cluster 2023-07-29T17:17:42.300575+0000 mgr.y (mgr.25169) 290 : cluster 0 pgmap v201: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:44.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:43 smithi078 bash[58614]: cluster 2023-07-29T17:17:42.300575+0000 mgr.y (mgr.25169) 290 : cluster 0 pgmap v201: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:44.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:43 smithi078 bash[60360]: cluster 2023-07-29T17:17:42.300575+0000 mgr.y (mgr.25169) 290 : cluster 0 pgmap v201: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:46.047 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:45 smithi129 bash[53756]: cluster 2023-07-29T17:17:44.302282+0000 mgr.y (mgr.25169) 291 : cluster 0 pgmap v202: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:46.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:45 smithi078 bash[58614]: cluster 2023-07-29T17:17:44.302282+0000 mgr.y (mgr.25169) 291 : cluster 0 pgmap v202: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:46.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:45 smithi078 bash[60360]: cluster 2023-07-29T17:17:44.302282+0000 mgr.y (mgr.25169) 291 : cluster 0 pgmap v202: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:46.424 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:46 smithi129 bash[60866]: debug 2023-07-29T17:17:46.024+0000 7f95b3dfc640 -1 osd.6 108 log_to_monitors true 2023-07-29T17:17:47.175 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:17:46 smithi129 bash[60866]: debug 2023-07-29T17:17:46.840+0000 7f95a3a66700 -1 osd.6 108 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:17:47.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:46 smithi129 bash[53756]: audit 2023-07-29T17:17:46.029236+0000 mon.a (mon.0) 653 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:17:47.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:46 smithi129 bash[53756]: audit 2023-07-29T17:17:46.030841+0000 mon.b (mon.1) 7 : audit 1 from='osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:17:47.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:46 smithi129 bash[53756]: audit 2023-07-29T17:17:46.544746+0000 mon.a (mon.0) 654 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:47.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:46 smithi129 bash[53756]: audit 2023-07-29T17:17:46.555289+0000 mon.a (mon.0) 655 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:47.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[58614]: audit 2023-07-29T17:17:46.029236+0000 mon.a (mon.0) 653 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:17:47.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[58614]: audit 2023-07-29T17:17:46.030841+0000 mon.b (mon.1) 7 : audit 1 from='osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:17:47.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[58614]: audit 2023-07-29T17:17:46.544746+0000 mon.a (mon.0) 654 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:47.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[58614]: audit 2023-07-29T17:17:46.555289+0000 mon.a (mon.0) 655 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:47.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[60360]: audit 2023-07-29T17:17:46.029236+0000 mon.a (mon.0) 653 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:17:47.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[60360]: audit 2023-07-29T17:17:46.030841+0000 mon.b (mon.1) 7 : audit 1 from='osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T17:17:47.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[60360]: audit 2023-07-29T17:17:46.544746+0000 mon.a (mon.0) 654 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:47.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:46 smithi078 bash[60360]: audit 2023-07-29T17:17:46.555289+0000 mon.a (mon.0) 655 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:48.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: cluster 2023-07-29T17:17:46.303540+0000 mgr.y (mgr.25169) 292 : cluster 0 pgmap v203: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:48.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: cluster 2023-07-29T17:17:46.781229+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e111 with expected crc 2023-07-29T17:17:48.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: audit 2023-07-29T17:17:46.784322+0000 mon.a (mon.0) 656 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T17:17:48.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: cluster 2023-07-29T17:17:46.784481+0000 mon.a (mon.0) 657 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-07-29T17:17:48.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: audit 2023-07-29T17:17:46.786953+0000 mon.a (mon.0) 658 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:48.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: audit 2023-07-29T17:17:46.788575+0000 mon.b (mon.1) 8 : audit 1 from='osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:48.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: audit 2023-07-29T17:17:47.357697+0000 mon.a (mon.0) 659 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:48.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:47 smithi129 bash[53756]: audit 2023-07-29T17:17:47.368683+0000 mon.a (mon.0) 660 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:48.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: cluster 2023-07-29T17:17:46.303540+0000 mgr.y (mgr.25169) 292 : cluster 0 pgmap v203: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:48.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: cluster 2023-07-29T17:17:46.781229+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e111 with expected crc 2023-07-29T17:17:48.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: audit 2023-07-29T17:17:46.784322+0000 mon.a (mon.0) 656 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T17:17:48.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: cluster 2023-07-29T17:17:46.784481+0000 mon.a (mon.0) 657 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-07-29T17:17:48.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: audit 2023-07-29T17:17:46.786953+0000 mon.a (mon.0) 658 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:48.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: audit 2023-07-29T17:17:46.788575+0000 mon.b (mon.1) 8 : audit 1 from='osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:48.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: audit 2023-07-29T17:17:47.357697+0000 mon.a (mon.0) 659 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:48.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[58614]: audit 2023-07-29T17:17:47.368683+0000 mon.a (mon.0) 660 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:48.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: cluster 2023-07-29T17:17:46.303540+0000 mgr.y (mgr.25169) 292 : cluster 0 pgmap v203: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2023-07-29T17:17:48.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: cluster 2023-07-29T17:17:46.781229+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e111 with expected crc 2023-07-29T17:17:48.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: audit 2023-07-29T17:17:46.784322+0000 mon.a (mon.0) 656 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T17:17:48.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: cluster 2023-07-29T17:17:46.784481+0000 mon.a (mon.0) 657 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-07-29T17:17:48.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: audit 2023-07-29T17:17:46.786953+0000 mon.a (mon.0) 658 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:48.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: audit 2023-07-29T17:17:46.788575+0000 mon.b (mon.1) 8 : audit 1 from='osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:17:48.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: audit 2023-07-29T17:17:47.357697+0000 mon.a (mon.0) 659 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:48.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:47 smithi078 bash[60360]: audit 2023-07-29T17:17:47.368683+0000 mon.a (mon.0) 660 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.778261+0000 mon.a (mon.0) 661 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:17:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.788056+0000 mon.a (mon.0) 662 : cluster 1 osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250] boot 2023-07-29T17:17:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.788156+0000 mon.a (mon.0) 663 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2023-07-29T17:17:49.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: audit 2023-07-29T17:17:47.789941+0000 mon.a (mon.0) 664 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:17:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.791980+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.794869+0000 osd.7 (osd.7) 60 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.798929+0000 osd.7 (osd.7) 61 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.802005+0000 osd.7 (osd.7) 62 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:47.804422+0000 osd.7 (osd.7) 63 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: audit 2023-07-29T17:17:48.541248+0000 mon.a (mon.0) 665 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:49.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.750756+0000 mon.a (mon.0) 666 : cluster 3 Health check update: Degraded data redundancy: 124/999 objects degraded (12.412%), 18 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:49.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:48 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.789181+0000 mon.a (mon.0) 667 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-07-29T17:17:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.778261+0000 mon.a (mon.0) 661 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:17:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.788056+0000 mon.a (mon.0) 662 : cluster 1 osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250] boot 2023-07-29T17:17:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.788156+0000 mon.a (mon.0) 663 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2023-07-29T17:17:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: audit 2023-07-29T17:17:47.789941+0000 mon.a (mon.0) 664 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:17:49.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.791980+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.794869+0000 osd.7 (osd.7) 60 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.798929+0000 osd.7 (osd.7) 61 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.802005+0000 osd.7 (osd.7) 62 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:47.804422+0000 osd.7 (osd.7) 63 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: audit 2023-07-29T17:17:48.541248+0000 mon.a (mon.0) 665 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.750756+0000 mon.a (mon.0) 666 : cluster 3 Health check update: Degraded data redundancy: 124/999 objects degraded (12.412%), 18 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:49.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.789181+0000 mon.a (mon.0) 667 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-07-29T17:17:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.778261+0000 mon.a (mon.0) 661 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:17:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.788056+0000 mon.a (mon.0) 662 : cluster 1 osd.6 [v2:172.21.15.129:6816/2477754250,v1:172.21.15.129:6817/2477754250] boot 2023-07-29T17:17:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.788156+0000 mon.a (mon.0) 663 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2023-07-29T17:17:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: audit 2023-07-29T17:17:47.789941+0000 mon.a (mon.0) 664 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T17:17:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.791980+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.794869+0000 osd.7 (osd.7) 60 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.798929+0000 osd.7 (osd.7) 61 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.802005+0000 osd.7 (osd.7) 62 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:47.804422+0000 osd.7 (osd.7) 63 : cluster 3 failed to encode map e112 with expected crc 2023-07-29T17:17:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: audit 2023-07-29T17:17:48.541248+0000 mon.a (mon.0) 665 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.750756+0000 mon.a (mon.0) 666 : cluster 3 Health check update: Degraded data redundancy: 124/999 objects degraded (12.412%), 18 pgs degraded (PG_DEGRADED) 2023-07-29T17:17:49.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:48 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.789181+0000 mon.a (mon.0) 667 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-07-29T17:17:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:49 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.304982+0000 mgr.y (mgr.25169) 293 : cluster 0 pgmap v206: 161 pgs: 19 peering, 12 active+undersized+degraded, 22 active+undersized, 108 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 81/999 objects degraded (8.108%) 2023-07-29T17:17:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:49 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.786770+0000 osd.7 (osd.7) 64 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:49 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.793211+0000 osd.7 (osd.7) 65 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:49 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.797816+0000 osd.7 (osd.7) 66 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:49 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.799690+0000 osd.7 (osd.7) 67 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:49 smithi129 bash[53756]: cluster 2023-07-29T17:17:48.801334+0000 osd.7 (osd.7) 68 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.304982+0000 mgr.y (mgr.25169) 293 : cluster 0 pgmap v206: 161 pgs: 19 peering, 12 active+undersized+degraded, 22 active+undersized, 108 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 81/999 objects degraded (8.108%) 2023-07-29T17:17:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.786770+0000 osd.7 (osd.7) 64 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.793211+0000 osd.7 (osd.7) 65 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.797816+0000 osd.7 (osd.7) 66 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.799690+0000 osd.7 (osd.7) 67 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[58614]: cluster 2023-07-29T17:17:48.801334+0000 osd.7 (osd.7) 68 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.304982+0000 mgr.y (mgr.25169) 293 : cluster 0 pgmap v206: 161 pgs: 19 peering, 12 active+undersized+degraded, 22 active+undersized, 108 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 81/999 objects degraded (8.108%) 2023-07-29T17:17:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.786770+0000 osd.7 (osd.7) 64 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.793211+0000 osd.7 (osd.7) 65 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.797816+0000 osd.7 (osd.7) 66 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.799690+0000 osd.7 (osd.7) 67 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:49 smithi078 bash[60360]: cluster 2023-07-29T17:17:48.801334+0000 osd.7 (osd.7) 68 : cluster 3 failed to encode map e113 with expected crc 2023-07-29T17:17:52.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:51 smithi129 bash[53756]: cluster 2023-07-29T17:17:50.306302+0000 mgr.y (mgr.25169) 294 : cluster 0 pgmap v208: 161 pgs: 19 peering, 7 active+undersized+degraded, 13 active+undersized, 122 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 45/999 objects degraded (4.505%) 2023-07-29T17:17:52.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:51 smithi078 bash[58614]: cluster 2023-07-29T17:17:50.306302+0000 mgr.y (mgr.25169) 294 : cluster 0 pgmap v208: 161 pgs: 19 peering, 7 active+undersized+degraded, 13 active+undersized, 122 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 45/999 objects degraded (4.505%) 2023-07-29T17:17:52.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:51 smithi078 bash[60360]: cluster 2023-07-29T17:17:50.306302+0000 mgr.y (mgr.25169) 294 : cluster 0 pgmap v208: 161 pgs: 19 peering, 7 active+undersized+degraded, 13 active+undersized, 122 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 45/999 objects degraded (4.505%) 2023-07-29T17:17:53.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:52 smithi129 bash[53756]: cluster 2023-07-29T17:17:52.817449+0000 mon.a (mon.0) 668 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/999 objects degraded (4.505%), 7 pgs degraded) 2023-07-29T17:17:53.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:52 smithi078 bash[58614]: cluster 2023-07-29T17:17:52.817449+0000 mon.a (mon.0) 668 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/999 objects degraded (4.505%), 7 pgs degraded) 2023-07-29T17:17:53.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:52 smithi078 bash[60360]: cluster 2023-07-29T17:17:52.817449+0000 mon.a (mon.0) 668 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/999 objects degraded (4.505%), 7 pgs degraded) 2023-07-29T17:17:54.174 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:53 smithi129 bash[53756]: cluster 2023-07-29T17:17:52.307297+0000 mgr.y (mgr.25169) 295 : cluster 0 pgmap v209: 161 pgs: 19 peering, 142 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:54.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:53 smithi078 bash[58614]: cluster 2023-07-29T17:17:52.307297+0000 mgr.y (mgr.25169) 295 : cluster 0 pgmap v209: 161 pgs: 19 peering, 142 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:54.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:53 smithi078 bash[60360]: cluster 2023-07-29T17:17:52.307297+0000 mgr.y (mgr.25169) 295 : cluster 0 pgmap v209: 161 pgs: 19 peering, 142 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:56.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:55 smithi129 bash[53756]: cluster 2023-07-29T17:17:54.308716+0000 mgr.y (mgr.25169) 296 : cluster 0 pgmap v210: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:56.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:55 smithi078 bash[58614]: cluster 2023-07-29T17:17:54.308716+0000 mgr.y (mgr.25169) 296 : cluster 0 pgmap v210: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:56.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:55 smithi078 bash[60360]: cluster 2023-07-29T17:17:54.308716+0000 mgr.y (mgr.25169) 296 : cluster 0 pgmap v210: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:58.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: cluster 2023-07-29T17:17:56.309594+0000 mgr.y (mgr.25169) 297 : cluster 0 pgmap v211: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:58.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.627126+0000 mon.a (mon.0) 669 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.638377+0000 mon.a (mon.0) 670 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.640485+0000 mon.a (mon.0) 671 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:58.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.641999+0000 mon.a (mon.0) 672 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:58.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.652126+0000 mon.a (mon.0) 673 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.739284+0000 mon.a (mon.0) 674 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:58.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.743254+0000 mon.a (mon.0) 675 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.745556+0000 mon.a (mon.0) 676 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.747203+0000 mon.a (mon.0) 677 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:57 smithi129 bash[53756]: audit 2023-07-29T17:17:57.749578+0000 mon.a (mon.0) 678 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-29T17:17:58.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: cluster 2023-07-29T17:17:56.309594+0000 mgr.y (mgr.25169) 297 : cluster 0 pgmap v211: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:58.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.627126+0000 mon.a (mon.0) 669 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.638377+0000 mon.a (mon.0) 670 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.640485+0000 mon.a (mon.0) 671 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:58.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.641999+0000 mon.a (mon.0) 672 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:58.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.652126+0000 mon.a (mon.0) 673 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.739284+0000 mon.a (mon.0) 674 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:58.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.743254+0000 mon.a (mon.0) 675 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.745556+0000 mon.a (mon.0) 676 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.747203+0000 mon.a (mon.0) 677 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[58614]: audit 2023-07-29T17:17:57.749578+0000 mon.a (mon.0) 678 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-29T17:17:58.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: cluster 2023-07-29T17:17:56.309594+0000 mgr.y (mgr.25169) 297 : cluster 0 pgmap v211: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.627126+0000 mon.a (mon.0) 669 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.638377+0000 mon.a (mon.0) 670 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.640485+0000 mon.a (mon.0) 671 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.641999+0000 mon.a (mon.0) 672 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.652126+0000 mon.a (mon.0) 673 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.739284+0000 mon.a (mon.0) 674 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:17:58.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.743254+0000 mon.a (mon.0) 675 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.745556+0000 mon.a (mon.0) 676 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.747203+0000 mon.a (mon.0) 677 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:17:58.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:57 smithi078 bash[60360]: audit 2023-07-29T17:17:57.749578+0000 mon.a (mon.0) 678 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-29T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:58 smithi129 bash[53756]: cephadm 2023-07-29T17:17:57.643350+0000 mgr.y (mgr.25169) 298 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:58 smithi129 bash[53756]: cluster 2023-07-29T17:17:57.644757+0000 mgr.y (mgr.25169) 299 : cluster 0 pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:58 smithi129 bash[53756]: cluster 2023-07-29T17:17:57.645237+0000 mgr.y (mgr.25169) 300 : cluster 0 pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:58 smithi129 bash[53756]: audit 2023-07-29T17:17:57.749962+0000 mgr.y (mgr.25169) 301 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-29T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:58 smithi129 bash[53756]: cephadm 2023-07-29T17:17:57.751829+0000 mgr.y (mgr.25169) 302 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-07-29T17:17:59.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:17:58 smithi129 bash[53756]: audit 2023-07-29T17:17:58.564187+0000 mon.a (mon.0) 679 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:59.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[58614]: cephadm 2023-07-29T17:17:57.643350+0000 mgr.y (mgr.25169) 298 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:59.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[58614]: cluster 2023-07-29T17:17:57.644757+0000 mgr.y (mgr.25169) 299 : cluster 0 pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:59.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[58614]: cluster 2023-07-29T17:17:57.645237+0000 mgr.y (mgr.25169) 300 : cluster 0 pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:59.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[58614]: audit 2023-07-29T17:17:57.749962+0000 mgr.y (mgr.25169) 301 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-29T17:17:59.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[58614]: cephadm 2023-07-29T17:17:57.751829+0000 mgr.y (mgr.25169) 302 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-07-29T17:17:59.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[58614]: audit 2023-07-29T17:17:58.564187+0000 mon.a (mon.0) 679 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:59.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[60360]: cephadm 2023-07-29T17:17:57.643350+0000 mgr.y (mgr.25169) 298 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:17:59.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[60360]: cluster 2023-07-29T17:17:57.644757+0000 mgr.y (mgr.25169) 299 : cluster 0 pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:59.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[60360]: cluster 2023-07-29T17:17:57.645237+0000 mgr.y (mgr.25169) 300 : cluster 0 pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:17:59.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[60360]: audit 2023-07-29T17:17:57.749962+0000 mgr.y (mgr.25169) 301 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-29T17:17:59.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[60360]: cephadm 2023-07-29T17:17:57.751829+0000 mgr.y (mgr.25169) 302 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-07-29T17:17:59.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:17:58 smithi078 bash[60360]: audit 2023-07-29T17:17:58.564187+0000 mon.a (mon.0) 679 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:17:59.729 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:18:00.467 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:18:00.467 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7m) 102s ago 14m 10.7M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (7m) 13s ago 13m 26.3M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 13s ago 11m - - 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (6m) 13s ago 18m 430M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (7m) 102s ago 20m 473M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5m) 102s ago 20m 64.3M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (4m) 13s ago 18m 53.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:18:00.468 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4m) 102s ago 18m 48.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (8m) 102s ago 14m 8844k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (7m) 13s ago 14m 3812k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3m) 102s ago 18m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2m) 102s ago 17m 52.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2m) 102s ago 17m 46.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (107s) 102s ago 16m - 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:18:00.469 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (77s) 13s ago 16m 52.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:18:00.471 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (48s) 13s ago 16m 49.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:18:00.471 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (19s) 13s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:18:00.471 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (15m) 13s ago 15m 67.3M 4096M 16.2.0 24ecd6d5f14c 682d54a619a5 2023-07-29T17:18:00.471 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 13s ago 14m - - 2023-07-29T17:18:00.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (12m) 102s ago 12m 55.8M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:18:00.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (12m) 13s ago 12m 56.3M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:18:00.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (11m) 102s ago 11m 56.0M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:18:00.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (11m) 13s ago 11m 56.2M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:18:00.908 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:18:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:18:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:18:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:18:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:18:00.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 7 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:00.910 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:18:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2023-07-29T17:18:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 12 2023-07-29T17:18:00.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:18:00.911 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:18:01.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:18:01.276 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:18:01.276 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:18:01.276 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:18:01.276 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:18:01.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:18:01.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:18:01.277 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:18:01.277 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "12/23 daemons upgraded", 2023-07-29T17:18:01.277 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2023-07-29T17:18:01.277 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:18:01.277 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:18:01.508 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: cluster 2023-07-29T17:17:59.646583+0000 mgr.y (mgr.25169) 303 : cluster 0 pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:01.509 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: audit 2023-07-29T17:17:59.693299+0000 mgr.y (mgr.25169) 304 : audit 0 from='client.44429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:01.509 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: audit 2023-07-29T17:18:00.064864+0000 mgr.y (mgr.25169) 305 : audit 0 from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:01.509 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: cephadm 2023-07-29T17:18:00.172999+0000 mgr.y (mgr.25169) 306 : cephadm 1 Upgrade: Updating osd.7 2023-07-29T17:18:01.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: audit 2023-07-29T17:18:00.181498+0000 mon.a (mon.0) 680 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:01.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: audit 2023-07-29T17:18:00.183216+0000 mon.a (mon.0) 681 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:18:01.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: audit 2023-07-29T17:18:00.184260+0000 mon.a (mon.0) 682 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:01.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: cephadm 2023-07-29T17:18:00.187422+0000 mgr.y (mgr.25169) 307 : cephadm 1 Deploying daemon osd.7 on smithi129 2023-07-29T17:18:01.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:01 smithi129 bash[53756]: audit 2023-07-29T17:18:00.885788+0000 mon.c (mon.2) 11 : audit 0 from='client.34617 172.21.15.78:0/521580969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:01.678 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:18:01.679 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:18:01.679 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:18:01.679 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:18:01.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: cluster 2023-07-29T17:17:59.646583+0000 mgr.y (mgr.25169) 303 : cluster 0 pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:01.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: audit 2023-07-29T17:17:59.693299+0000 mgr.y (mgr.25169) 304 : audit 0 from='client.44429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:01.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: audit 2023-07-29T17:18:00.064864+0000 mgr.y (mgr.25169) 305 : audit 0 from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:01.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: cephadm 2023-07-29T17:18:00.172999+0000 mgr.y (mgr.25169) 306 : cephadm 1 Upgrade: Updating osd.7 2023-07-29T17:18:01.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: audit 2023-07-29T17:18:00.181498+0000 mon.a (mon.0) 680 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:01.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: audit 2023-07-29T17:18:00.183216+0000 mon.a (mon.0) 681 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:18:01.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: audit 2023-07-29T17:18:00.184260+0000 mon.a (mon.0) 682 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:01.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: cephadm 2023-07-29T17:18:00.187422+0000 mgr.y (mgr.25169) 307 : cephadm 1 Deploying daemon osd.7 on smithi129 2023-07-29T17:18:01.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[60360]: audit 2023-07-29T17:18:00.885788+0000 mon.c (mon.2) 11 : audit 0 from='client.34617 172.21.15.78:0/521580969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:01.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: cluster 2023-07-29T17:17:59.646583+0000 mgr.y (mgr.25169) 303 : cluster 0 pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: audit 2023-07-29T17:17:59.693299+0000 mgr.y (mgr.25169) 304 : audit 0 from='client.44429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: audit 2023-07-29T17:18:00.064864+0000 mgr.y (mgr.25169) 305 : audit 0 from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: cephadm 2023-07-29T17:18:00.172999+0000 mgr.y (mgr.25169) 306 : cephadm 1 Upgrade: Updating osd.7 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: audit 2023-07-29T17:18:00.181498+0000 mon.a (mon.0) 680 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: audit 2023-07-29T17:18:00.183216+0000 mon.a (mon.0) 681 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: audit 2023-07-29T17:18:00.184260+0000 mon.a (mon.0) 682 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:01.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: cephadm 2023-07-29T17:18:00.187422+0000 mgr.y (mgr.25169) 307 : cephadm 1 Deploying daemon osd.7 on smithi129 2023-07-29T17:18:01.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:01 smithi078 bash[58614]: audit 2023-07-29T17:18:00.885788+0000 mon.c (mon.2) 11 : audit 0 from='client.34617 172.21.15.78:0/521580969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:02.490 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:02 smithi129 bash[53756]: audit 2023-07-29T17:18:00.428243+0000 mgr.y (mgr.25169) 308 : audit 0 from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:02.490 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:02 smithi129 bash[53756]: audit 2023-07-29T17:18:01.656389+0000 mon.c (mon.2) 12 : audit 0 from='client.? 172.21.15.78:0/3225671842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:18:02.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:02 smithi078 bash[58614]: audit 2023-07-29T17:18:00.428243+0000 mgr.y (mgr.25169) 308 : audit 0 from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:02.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:02 smithi078 bash[58614]: audit 2023-07-29T17:18:01.656389+0000 mon.c (mon.2) 12 : audit 0 from='client.? 172.21.15.78:0/3225671842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:18:02.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:02 smithi078 bash[60360]: audit 2023-07-29T17:18:00.428243+0000 mgr.y (mgr.25169) 308 : audit 0 from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:02.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:02 smithi078 bash[60360]: audit 2023-07-29T17:18:01.656389+0000 mon.c (mon.2) 12 : audit 0 from='client.? 172.21.15.78:0/3225671842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:18:03.562 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:03 smithi129 bash[53756]: audit 2023-07-29T17:18:01.252430+0000 mgr.y (mgr.25169) 309 : audit 0 from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:03.563 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:03 smithi129 bash[53756]: cluster 2023-07-29T17:18:01.647197+0000 mgr.y (mgr.25169) 310 : cluster 0 pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:03.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:03 smithi078 bash[58614]: audit 2023-07-29T17:18:01.252430+0000 mgr.y (mgr.25169) 309 : audit 0 from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:03.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:03 smithi078 bash[58614]: cluster 2023-07-29T17:18:01.647197+0000 mgr.y (mgr.25169) 310 : cluster 0 pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:03.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:03 smithi078 bash[60360]: audit 2023-07-29T17:18:01.252430+0000 mgr.y (mgr.25169) 309 : audit 0 from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:03.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:03 smithi078 bash[60360]: cluster 2023-07-29T17:18:01.647197+0000 mgr.y (mgr.25169) 310 : cluster 0 pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:05.044 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:04 smithi129 systemd[1]: Stopping Ceph osd.7 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:18:05.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[53756]: cluster 2023-07-29T17:18:03.648560+0000 mgr.y (mgr.25169) 311 : cluster 0 pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:05.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[53756]: cluster 2023-07-29T17:18:05.072325+0000 mon.a (mon.0) 683 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-29T17:18:05.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[53756]: cluster 2023-07-29T17:18:05.072437+0000 mon.a (mon.0) 684 : cluster 1 osd.7 failed (root=default,host=smithi129) (connection refused reported by osd.1) 2023-07-29T17:18:05.425 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[34912]: debug 2023-07-29T17:18:05.020+0000 7f46f8ad1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:18:05.426 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[34912]: debug 2023-07-29T17:18:05.020+0000 7f46f8ad1700 -1 osd.7 113 *** Got signal Terminated *** 2023-07-29T17:18:05.426 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[34912]: debug 2023-07-29T17:18:05.020+0000 7f46f8ad1700 -1 osd.7 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:18:05.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:05 smithi078 bash[58614]: cluster 2023-07-29T17:18:03.648560+0000 mgr.y (mgr.25169) 311 : cluster 0 pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:05.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:05 smithi078 bash[58614]: cluster 2023-07-29T17:18:05.072325+0000 mon.a (mon.0) 683 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-29T17:18:05.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:05 smithi078 bash[58614]: cluster 2023-07-29T17:18:05.072437+0000 mon.a (mon.0) 684 : cluster 1 osd.7 failed (root=default,host=smithi129) (connection refused reported by osd.1) 2023-07-29T17:18:05.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:05 smithi078 bash[60360]: cluster 2023-07-29T17:18:03.648560+0000 mgr.y (mgr.25169) 311 : cluster 0 pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:05.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:05 smithi078 bash[60360]: cluster 2023-07-29T17:18:05.072325+0000 mon.a (mon.0) 683 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-29T17:18:05.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:05 smithi078 bash[60360]: cluster 2023-07-29T17:18:05.072437+0000 mon.a (mon.0) 684 : cluster 1 osd.7 failed (root=default,host=smithi129) (connection refused reported by osd.1) 2023-07-29T17:18:06.225 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:05 smithi129 bash[62499]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd.7 2023-07-29T17:18:06.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:06 smithi129 bash[53756]: cluster 2023-07-29T17:18:05.198808+0000 mon.a (mon.0) 685 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:18:06.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:06 smithi129 bash[53756]: cluster 2023-07-29T17:18:05.198875+0000 mon.a (mon.0) 686 : cluster 3 Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2023-07-29T17:18:06.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:06 smithi129 bash[53756]: cluster 2023-07-29T17:18:05.211387+0000 mon.a (mon.0) 687 : cluster 0 osdmap e114: 8 total, 7 up, 8 in 2023-07-29T17:18:06.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:06 smithi078 bash[58614]: cluster 2023-07-29T17:18:05.198808+0000 mon.a (mon.0) 685 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:18:06.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:06 smithi078 bash[58614]: cluster 2023-07-29T17:18:05.198875+0000 mon.a (mon.0) 686 : cluster 3 Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2023-07-29T17:18:06.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:06 smithi078 bash[58614]: cluster 2023-07-29T17:18:05.211387+0000 mon.a (mon.0) 687 : cluster 0 osdmap e114: 8 total, 7 up, 8 in 2023-07-29T17:18:06.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:06 smithi078 bash[60360]: cluster 2023-07-29T17:18:05.198808+0000 mon.a (mon.0) 685 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-29T17:18:06.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:06 smithi078 bash[60360]: cluster 2023-07-29T17:18:05.198875+0000 mon.a (mon.0) 686 : cluster 3 Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2023-07-29T17:18:06.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:06 smithi078 bash[60360]: cluster 2023-07-29T17:18:05.211387+0000 mon.a (mon.0) 687 : cluster 0 osdmap e114: 8 total, 7 up, 8 in 2023-07-29T17:18:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:07 smithi129 bash[53756]: cluster 2023-07-29T17:18:05.649307+0000 mgr.y (mgr.25169) 312 : cluster 0 pgmap v218: 161 pgs: 22 stale+active+clean, 139 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:07.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:07 smithi129 bash[53756]: cluster 2023-07-29T17:18:06.224386+0000 mon.a (mon.0) 688 : cluster 0 osdmap e115: 8 total, 7 up, 8 in 2023-07-29T17:18:07.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:07 smithi078 bash[60360]: cluster 2023-07-29T17:18:05.649307+0000 mgr.y (mgr.25169) 312 : cluster 0 pgmap v218: 161 pgs: 22 stale+active+clean, 139 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:07.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:07 smithi078 bash[60360]: cluster 2023-07-29T17:18:06.224386+0000 mon.a (mon.0) 688 : cluster 0 osdmap e115: 8 total, 7 up, 8 in 2023-07-29T17:18:07.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:07 smithi078 bash[58614]: cluster 2023-07-29T17:18:05.649307+0000 mgr.y (mgr.25169) 312 : cluster 0 pgmap v218: 161 pgs: 22 stale+active+clean, 139 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:07.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:07 smithi078 bash[58614]: cluster 2023-07-29T17:18:06.224386+0000 mon.a (mon.0) 688 : cluster 0 osdmap e115: 8 total, 7 up, 8 in 2023-07-29T17:18:08.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:08 smithi129 bash[53756]: cluster 2023-07-29T17:18:07.650475+0000 mgr.y (mgr.25169) 313 : cluster 0 pgmap v220: 161 pgs: 5 active+undersized+degraded, 14 stale+active+clean, 16 active+undersized, 126 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 27/999 objects degraded (2.703%) 2023-07-29T17:18:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:08 smithi129 bash[53756]: cluster 2023-07-29T17:18:08.203612+0000 mon.a (mon.0) 689 : cluster 3 Health check failed: Degraded data redundancy: 27/999 objects degraded (2.703%), 5 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:08.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:08 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.7.service: Succeeded. 2023-07-29T17:18:08.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:08 smithi129 systemd[1]: Stopped Ceph osd.7 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:18:08.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:08 smithi129 systemd[1]: Started Ceph osd.7 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:18:08.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:08 smithi078 bash[58614]: cluster 2023-07-29T17:18:07.650475+0000 mgr.y (mgr.25169) 313 : cluster 0 pgmap v220: 161 pgs: 5 active+undersized+degraded, 14 stale+active+clean, 16 active+undersized, 126 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 27/999 objects degraded (2.703%) 2023-07-29T17:18:08.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:08 smithi078 bash[58614]: cluster 2023-07-29T17:18:08.203612+0000 mon.a (mon.0) 689 : cluster 3 Health check failed: Degraded data redundancy: 27/999 objects degraded (2.703%), 5 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:08.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:08 smithi078 bash[60360]: cluster 2023-07-29T17:18:07.650475+0000 mgr.y (mgr.25169) 313 : cluster 0 pgmap v220: 161 pgs: 5 active+undersized+degraded, 14 stale+active+clean, 16 active+undersized, 126 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 27/999 objects degraded (2.703%) 2023-07-29T17:18:08.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:08 smithi078 bash[60360]: cluster 2023-07-29T17:18:08.203612+0000 mon.a (mon.0) 689 : cluster 3 Health check failed: Degraded data redundancy: 27/999 objects degraded (2.703%), 5 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:10.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:09 smithi129 bash[53756]: audit 2023-07-29T17:18:08.693523+0000 mon.a (mon.0) 690 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:10.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:09 smithi129 bash[53756]: audit 2023-07-29T17:18:08.703603+0000 mon.a (mon.0) 691 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:10.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:09 smithi078 bash[60360]: audit 2023-07-29T17:18:08.693523+0000 mon.a (mon.0) 690 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:10.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:09 smithi078 bash[60360]: audit 2023-07-29T17:18:08.703603+0000 mon.a (mon.0) 691 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:10.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:09 smithi078 bash[58614]: audit 2023-07-29T17:18:08.693523+0000 mon.a (mon.0) 690 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:10.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:09 smithi078 bash[58614]: audit 2023-07-29T17:18:08.703603+0000 mon.a (mon.0) 691 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:10.788 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-29T17:18:10.788 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-07-29T17:18:10.789 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-07-29T17:18:10.789 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-29T17:18:10.789 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2023-07-29T17:18:10.789 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-29T17:18:10.789 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[62773]: --> ceph-volume raw activate successful for osd ID: 7 2023-07-29T17:18:10.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:10 smithi129 bash[53756]: cluster 2023-07-29T17:18:09.652148+0000 mgr.y (mgr.25169) 314 : cluster 0 pgmap v221: 161 pgs: 20 active+undersized+degraded, 2 stale+active+clean, 37 active+undersized, 102 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 111/999 objects degraded (11.111%) 2023-07-29T17:18:11.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:10 smithi078 bash[60360]: cluster 2023-07-29T17:18:09.652148+0000 mgr.y (mgr.25169) 314 : cluster 0 pgmap v221: 161 pgs: 20 active+undersized+degraded, 2 stale+active+clean, 37 active+undersized, 102 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 111/999 objects degraded (11.111%) 2023-07-29T17:18:11.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:10 smithi078 bash[58614]: cluster 2023-07-29T17:18:09.652148+0000 mgr.y (mgr.25169) 314 : cluster 0 pgmap v221: 161 pgs: 20 active+undersized+degraded, 2 stale+active+clean, 37 active+undersized, 102 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 111/999 objects degraded (11.111%) 2023-07-29T17:18:12.675 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:12 smithi129 bash[63075]: debug 2023-07-29T17:18:12.307+0000 7fb063dc9700 1 -- 172.21.15.129:0/582640820 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56335b7124e0 con 0x56335b76d000 2023-07-29T17:18:13.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:12 smithi129 bash[53756]: cluster 2023-07-29T17:18:11.653445+0000 mgr.y (mgr.25169) 315 : cluster 0 pgmap v222: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:13.175 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:12 smithi129 bash[63075]: debug 2023-07-29T17:18:12.855+0000 7fb06cf4e640 -1 Falling back to public interface 2023-07-29T17:18:13.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:12 smithi078 bash[60360]: cluster 2023-07-29T17:18:11.653445+0000 mgr.y (mgr.25169) 315 : cluster 0 pgmap v222: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:13.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:12 smithi078 bash[58614]: cluster 2023-07-29T17:18:11.653445+0000 mgr.y (mgr.25169) 315 : cluster 0 pgmap v222: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:14.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:13 smithi129 bash[53756]: cluster 2023-07-29T17:18:13.758658+0000 mon.a (mon.0) 692 : cluster 3 Health check update: Degraded data redundancy: 128/999 objects degraded (12.813%), 25 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:14.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:13 smithi078 bash[58614]: cluster 2023-07-29T17:18:13.758658+0000 mon.a (mon.0) 692 : cluster 3 Health check update: Degraded data redundancy: 128/999 objects degraded (12.813%), 25 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:14.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:13 smithi078 bash[60360]: cluster 2023-07-29T17:18:13.758658+0000 mon.a (mon.0) 692 : cluster 3 Health check update: Degraded data redundancy: 128/999 objects degraded (12.813%), 25 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:15.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:14 smithi129 bash[53756]: cluster 2023-07-29T17:18:13.655148+0000 mgr.y (mgr.25169) 316 : cluster 0 pgmap v223: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:15.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:14 smithi078 bash[58614]: cluster 2023-07-29T17:18:13.655148+0000 mgr.y (mgr.25169) 316 : cluster 0 pgmap v223: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:15.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:14 smithi078 bash[60360]: cluster 2023-07-29T17:18:13.655148+0000 mgr.y (mgr.25169) 316 : cluster 0 pgmap v223: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:16.425 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:15 smithi129 bash[63075]: debug 2023-07-29T17:18:15.935+0000 7fb06cf4e640 -1 osd.7 113 log_to_monitors true 2023-07-29T17:18:17.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:16 smithi129 bash[53756]: cluster 2023-07-29T17:18:15.656585+0000 mgr.y (mgr.25169) 317 : cluster 0 pgmap v224: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:17.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:16 smithi129 bash[53756]: audit 2023-07-29T17:18:15.941894+0000 mon.a (mon.0) 693 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:18:17.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:16 smithi129 bash[53756]: audit 2023-07-29T17:18:15.943499+0000 mon.b (mon.1) 9 : audit 1 from='osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:18:17.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:16 smithi129 bash[53756]: audit 2023-07-29T17:18:16.325460+0000 mon.a (mon.0) 694 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:17.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:16 smithi129 bash[53756]: audit 2023-07-29T17:18:16.336657+0000 mon.a (mon.0) 695 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:17.176 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:18:16 smithi129 bash[63075]: debug 2023-07-29T17:18:16.843+0000 7fb05a3b3700 -1 osd.7 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T17:18:17.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[58614]: cluster 2023-07-29T17:18:15.656585+0000 mgr.y (mgr.25169) 317 : cluster 0 pgmap v224: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:17.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[58614]: audit 2023-07-29T17:18:15.941894+0000 mon.a (mon.0) 693 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:18:17.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[58614]: audit 2023-07-29T17:18:15.943499+0000 mon.b (mon.1) 9 : audit 1 from='osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:18:17.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[58614]: audit 2023-07-29T17:18:16.325460+0000 mon.a (mon.0) 694 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:17.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[58614]: audit 2023-07-29T17:18:16.336657+0000 mon.a (mon.0) 695 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:17.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[60360]: cluster 2023-07-29T17:18:15.656585+0000 mgr.y (mgr.25169) 317 : cluster 0 pgmap v224: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:17.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[60360]: audit 2023-07-29T17:18:15.941894+0000 mon.a (mon.0) 693 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:18:17.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[60360]: audit 2023-07-29T17:18:15.943499+0000 mon.b (mon.1) 9 : audit 1 from='osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T17:18:17.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[60360]: audit 2023-07-29T17:18:16.325460+0000 mon.a (mon.0) 694 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:17.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:16 smithi078 bash[60360]: audit 2023-07-29T17:18:16.336657+0000 mon.a (mon.0) 695 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:18.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:17 smithi129 bash[53756]: audit 2023-07-29T17:18:16.803338+0000 mon.a (mon.0) 696 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T17:18:18.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:17 smithi129 bash[53756]: cluster 2023-07-29T17:18:16.803563+0000 mon.a (mon.0) 697 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-07-29T17:18:18.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:17 smithi129 bash[53756]: audit 2023-07-29T17:18:16.808560+0000 mon.a (mon.0) 698 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:18:18.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:17 smithi129 bash[53756]: audit 2023-07-29T17:18:16.810253+0000 mon.b (mon.1) 10 : audit 1 from='osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:18:18.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:17 smithi129 bash[53756]: audit 2023-07-29T17:18:17.135116+0000 mon.a (mon.0) 699 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:18.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:17 smithi129 bash[53756]: audit 2023-07-29T17:18:17.145268+0000 mon.a (mon.0) 700 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:18.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[58614]: audit 2023-07-29T17:18:16.803338+0000 mon.a (mon.0) 696 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T17:18:18.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[58614]: cluster 2023-07-29T17:18:16.803563+0000 mon.a (mon.0) 697 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-07-29T17:18:18.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[58614]: audit 2023-07-29T17:18:16.808560+0000 mon.a (mon.0) 698 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:18:18.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[58614]: audit 2023-07-29T17:18:16.810253+0000 mon.b (mon.1) 10 : audit 1 from='osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:18:18.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[58614]: audit 2023-07-29T17:18:17.135116+0000 mon.a (mon.0) 699 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:18.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[58614]: audit 2023-07-29T17:18:17.145268+0000 mon.a (mon.0) 700 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:18.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[60360]: audit 2023-07-29T17:18:16.803338+0000 mon.a (mon.0) 696 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T17:18:18.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[60360]: cluster 2023-07-29T17:18:16.803563+0000 mon.a (mon.0) 697 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-07-29T17:18:18.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[60360]: audit 2023-07-29T17:18:16.808560+0000 mon.a (mon.0) 698 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:18:18.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[60360]: audit 2023-07-29T17:18:16.810253+0000 mon.b (mon.1) 10 : audit 1 from='osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2023-07-29T17:18:18.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[60360]: audit 2023-07-29T17:18:17.135116+0000 mon.a (mon.0) 699 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:18.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:17 smithi078 bash[60360]: audit 2023-07-29T17:18:17.145268+0000 mon.a (mon.0) 700 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:19.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: cluster 2023-07-29T17:18:17.657982+0000 mgr.y (mgr.25169) 318 : cluster 0 pgmap v226: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:19.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: cluster 2023-07-29T17:18:17.798699+0000 mon.a (mon.0) 701 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:18:19.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: cluster 2023-07-29T17:18:17.812920+0000 mon.a (mon.0) 702 : cluster 1 osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221] boot 2023-07-29T17:18:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: cluster 2023-07-29T17:18:17.813037+0000 mon.a (mon.0) 703 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2023-07-29T17:18:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: audit 2023-07-29T17:18:17.815535+0000 mon.a (mon.0) 704 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:18:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: audit 2023-07-29T17:18:18.597573+0000 mon.a (mon.0) 705 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:18 smithi129 bash[53756]: cluster 2023-07-29T17:18:18.809062+0000 mon.a (mon.0) 706 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2023-07-29T17:18:19.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: cluster 2023-07-29T17:18:17.657982+0000 mgr.y (mgr.25169) 318 : cluster 0 pgmap v226: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:19.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: cluster 2023-07-29T17:18:17.798699+0000 mon.a (mon.0) 701 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:18:19.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: cluster 2023-07-29T17:18:17.812920+0000 mon.a (mon.0) 702 : cluster 1 osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221] boot 2023-07-29T17:18:19.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: cluster 2023-07-29T17:18:17.813037+0000 mon.a (mon.0) 703 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2023-07-29T17:18:19.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: audit 2023-07-29T17:18:17.815535+0000 mon.a (mon.0) 704 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:18:19.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: audit 2023-07-29T17:18:18.597573+0000 mon.a (mon.0) 705 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:19.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[58614]: cluster 2023-07-29T17:18:18.809062+0000 mon.a (mon.0) 706 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2023-07-29T17:18:19.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: cluster 2023-07-29T17:18:17.657982+0000 mgr.y (mgr.25169) 318 : cluster 0 pgmap v226: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2023-07-29T17:18:19.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: cluster 2023-07-29T17:18:17.798699+0000 mon.a (mon.0) 701 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-29T17:18:19.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: cluster 2023-07-29T17:18:17.812920+0000 mon.a (mon.0) 702 : cluster 1 osd.7 [v2:172.21.15.129:6824/2070228221,v1:172.21.15.129:6825/2070228221] boot 2023-07-29T17:18:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: cluster 2023-07-29T17:18:17.813037+0000 mon.a (mon.0) 703 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2023-07-29T17:18:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: audit 2023-07-29T17:18:17.815535+0000 mon.a (mon.0) 704 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T17:18:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: audit 2023-07-29T17:18:18.597573+0000 mon.a (mon.0) 705 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:18 smithi078 bash[60360]: cluster 2023-07-29T17:18:18.809062+0000 mon.a (mon.0) 706 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2023-07-29T17:18:20.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:19 smithi129 bash[53756]: cluster 2023-07-29T17:18:19.830978+0000 mon.a (mon.0) 707 : cluster 3 Health check update: Degraded data redundancy: 87/999 objects degraded (8.709%), 15 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:20.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:19 smithi078 bash[58614]: cluster 2023-07-29T17:18:19.830978+0000 mon.a (mon.0) 707 : cluster 3 Health check update: Degraded data redundancy: 87/999 objects degraded (8.709%), 15 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:20.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:19 smithi078 bash[60360]: cluster 2023-07-29T17:18:19.830978+0000 mon.a (mon.0) 707 : cluster 3 Health check update: Degraded data redundancy: 87/999 objects degraded (8.709%), 15 pgs degraded (PG_DEGRADED) 2023-07-29T17:18:21.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:20 smithi129 bash[53756]: cluster 2023-07-29T17:18:19.659113+0000 mgr.y (mgr.25169) 319 : cluster 0 pgmap v229: 161 pgs: 16 peering, 15 active+undersized+degraded, 34 active+undersized, 96 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 87/999 objects degraded (8.709%) 2023-07-29T17:18:21.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:20 smithi078 bash[60360]: cluster 2023-07-29T17:18:19.659113+0000 mgr.y (mgr.25169) 319 : cluster 0 pgmap v229: 161 pgs: 16 peering, 15 active+undersized+degraded, 34 active+undersized, 96 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 87/999 objects degraded (8.709%) 2023-07-29T17:18:21.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:20 smithi078 bash[58614]: cluster 2023-07-29T17:18:19.659113+0000 mgr.y (mgr.25169) 319 : cluster 0 pgmap v229: 161 pgs: 16 peering, 15 active+undersized+degraded, 34 active+undersized, 96 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 87/999 objects degraded (8.709%) 2023-07-29T17:18:23.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:22 smithi129 bash[53756]: cluster 2023-07-29T17:18:21.660266+0000 mgr.y (mgr.25169) 320 : cluster 0 pgmap v230: 161 pgs: 16 peering, 12 active+undersized+degraded, 27 active+undersized, 106 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-29T17:18:23.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:22 smithi078 bash[58614]: cluster 2023-07-29T17:18:21.660266+0000 mgr.y (mgr.25169) 320 : cluster 0 pgmap v230: 161 pgs: 16 peering, 12 active+undersized+degraded, 27 active+undersized, 106 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-29T17:18:23.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:22 smithi078 bash[60360]: cluster 2023-07-29T17:18:21.660266+0000 mgr.y (mgr.25169) 320 : cluster 0 pgmap v230: 161 pgs: 16 peering, 12 active+undersized+degraded, 27 active+undersized, 106 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-29T17:18:25.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:24 smithi129 bash[53756]: cluster 2023-07-29T17:18:23.661770+0000 mgr.y (mgr.25169) 321 : cluster 0 pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:25.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:24 smithi129 bash[53756]: cluster 2023-07-29T17:18:23.890380+0000 mon.a (mon.0) 708 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/999 objects degraded (7.307%), 12 pgs degraded) 2023-07-29T17:18:25.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:24 smithi078 bash[60360]: cluster 2023-07-29T17:18:23.661770+0000 mgr.y (mgr.25169) 321 : cluster 0 pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:25.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:24 smithi078 bash[60360]: cluster 2023-07-29T17:18:23.890380+0000 mon.a (mon.0) 708 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/999 objects degraded (7.307%), 12 pgs degraded) 2023-07-29T17:18:25.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:24 smithi078 bash[58614]: cluster 2023-07-29T17:18:23.661770+0000 mgr.y (mgr.25169) 321 : cluster 0 pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:25.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:24 smithi078 bash[58614]: cluster 2023-07-29T17:18:23.890380+0000 mon.a (mon.0) 708 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/999 objects degraded (7.307%), 12 pgs degraded) 2023-07-29T17:18:26.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:26 smithi129 bash[53756]: cluster 2023-07-29T17:18:25.662825+0000 mgr.y (mgr.25169) 322 : cluster 0 pgmap v232: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:27.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:26 smithi078 bash[58614]: cluster 2023-07-29T17:18:25.662825+0000 mgr.y (mgr.25169) 322 : cluster 0 pgmap v232: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:27.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:26 smithi078 bash[60360]: cluster 2023-07-29T17:18:25.662825+0000 mgr.y (mgr.25169) 322 : cluster 0 pgmap v232: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:28.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.206551+0000 mon.a (mon.0) 709 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.215436+0000 mon.a (mon.0) 710 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.217396+0000 mon.a (mon.0) 711 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:28.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.219226+0000 mon.a (mon.0) 712 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.230862+0000 mon.a (mon.0) 713 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.314219+0000 mon.a (mon.0) 714 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.318845+0000 mon.a (mon.0) 715 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.321455+0000 mon.a (mon.0) 716 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.323541+0000 mon.a (mon.0) 717 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.327128+0000 mon.a (mon.0) 718 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.337365+0000 mon.a (mon.0) 719 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.339227+0000 mon.a (mon.0) 720 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.347250+0000 mon.a (mon.0) 721 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.348668+0000 mon.a (mon.0) 722 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.354954+0000 mon.a (mon.0) 723 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.356178+0000 mon.a (mon.0) 724 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.362851+0000 mon.a (mon.0) 725 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-07-29T17:18:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.364287+0000 mon.a (mon.0) 726 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.371330+0000 mon.a (mon.0) 727 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.372789+0000 mon.a (mon.0) 728 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.379216+0000 mon.a (mon.0) 729 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.380653+0000 mon.a (mon.0) 730 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.387260+0000 mon.a (mon.0) 731 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.388645+0000 mon.a (mon.0) 732 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-07-29T17:18:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.394901+0000 mon.a (mon.0) 733 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-07-29T17:18:28.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.396401+0000 mon.a (mon.0) 734 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-07-29T17:18:28.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.402773+0000 mon.a (mon.0) 735 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-07-29T17:18:28.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:28 smithi129 bash[53756]: audit 2023-07-29T17:18:27.406573+0000 mon.a (mon.0) 736 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2023-07-29T17:18:28.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.206551+0000 mon.a (mon.0) 709 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.215436+0000 mon.a (mon.0) 710 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.217396+0000 mon.a (mon.0) 711 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.219226+0000 mon.a (mon.0) 712 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.230862+0000 mon.a (mon.0) 713 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.314219+0000 mon.a (mon.0) 714 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.318845+0000 mon.a (mon.0) 715 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.321455+0000 mon.a (mon.0) 716 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.323541+0000 mon.a (mon.0) 717 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.327128+0000 mon.a (mon.0) 718 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.337365+0000 mon.a (mon.0) 719 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.339227+0000 mon.a (mon.0) 720 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-07-29T17:18:28.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.347250+0000 mon.a (mon.0) 721 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-07-29T17:18:28.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.348668+0000 mon.a (mon.0) 722 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-07-29T17:18:28.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.354954+0000 mon.a (mon.0) 723 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-07-29T17:18:28.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.356178+0000 mon.a (mon.0) 724 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-07-29T17:18:28.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.362851+0000 mon.a (mon.0) 725 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-07-29T17:18:28.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.364287+0000 mon.a (mon.0) 726 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-07-29T17:18:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.206551+0000 mon.a (mon.0) 709 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.215436+0000 mon.a (mon.0) 710 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.217396+0000 mon.a (mon.0) 711 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.219226+0000 mon.a (mon.0) 712 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:18:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.230862+0000 mon.a (mon.0) 713 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.314219+0000 mon.a (mon.0) 714 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.318845+0000 mon.a (mon.0) 715 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.321455+0000 mon.a (mon.0) 716 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.323541+0000 mon.a (mon.0) 717 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.327128+0000 mon.a (mon.0) 718 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.337365+0000 mon.a (mon.0) 719 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.339227+0000 mon.a (mon.0) 720 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-07-29T17:18:28.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.347250+0000 mon.a (mon.0) 721 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-07-29T17:18:28.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.348668+0000 mon.a (mon.0) 722 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-07-29T17:18:28.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.354954+0000 mon.a (mon.0) 723 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-07-29T17:18:28.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.356178+0000 mon.a (mon.0) 724 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-07-29T17:18:28.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.362851+0000 mon.a (mon.0) 725 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-07-29T17:18:28.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.364287+0000 mon.a (mon.0) 726 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-07-29T17:18:28.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.371330+0000 mon.a (mon.0) 727 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-07-29T17:18:28.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.372789+0000 mon.a (mon.0) 728 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-07-29T17:18:28.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.379216+0000 mon.a (mon.0) 729 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-07-29T17:18:28.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.380653+0000 mon.a (mon.0) 730 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-07-29T17:18:28.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.387260+0000 mon.a (mon.0) 731 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-07-29T17:18:28.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.388645+0000 mon.a (mon.0) 732 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-07-29T17:18:28.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.394901+0000 mon.a (mon.0) 733 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-07-29T17:18:28.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.396401+0000 mon.a (mon.0) 734 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-07-29T17:18:28.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.402773+0000 mon.a (mon.0) 735 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-07-29T17:18:28.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[60360]: audit 2023-07-29T17:18:27.406573+0000 mon.a (mon.0) 736 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2023-07-29T17:18:28.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.371330+0000 mon.a (mon.0) 727 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-07-29T17:18:28.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.372789+0000 mon.a (mon.0) 728 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-07-29T17:18:28.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.379216+0000 mon.a (mon.0) 729 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.380653+0000 mon.a (mon.0) 730 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.387260+0000 mon.a (mon.0) 731 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.388645+0000 mon.a (mon.0) 732 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.394901+0000 mon.a (mon.0) 733 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.396401+0000 mon.a (mon.0) 734 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.402773+0000 mon.a (mon.0) 735 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-07-29T17:18:28.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:28 smithi078 bash[58614]: audit 2023-07-29T17:18:27.406573+0000 mon.a (mon.0) 736 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2023-07-29T17:18:29.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cephadm 2023-07-29T17:18:27.221001+0000 mgr.y (mgr.25169) 323 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:18:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cluster 2023-07-29T17:18:27.222536+0000 mgr.y (mgr.25169) 324 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cluster 2023-07-29T17:18:27.222907+0000 mgr.y (mgr.25169) 325 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cephadm 2023-07-29T17:18:27.328391+0000 mgr.y (mgr.25169) 326 : cephadm 1 Upgrade: Setting container_image for all osd 2023-07-29T17:18:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cephadm 2023-07-29T17:18:27.405956+0000 mgr.y (mgr.25169) 327 : cephadm 1 Upgrade: Setting require_osd_release to 18 reef 2023-07-29T17:18:29.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cluster 2023-07-29T17:18:28.398473+0000 mon.a (mon.0) 737 : cluster 1 Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2023-07-29T17:18:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: audit 2023-07-29T17:18:28.407932+0000 mon.a (mon.0) 738 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2023-07-29T17:18:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: cluster 2023-07-29T17:18:28.408057+0000 mon.a (mon.0) 739 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2023-07-29T17:18:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: audit 2023-07-29T17:18:28.412704+0000 mon.a (mon.0) 740 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: audit 2023-07-29T17:18:28.419700+0000 mon.a (mon.0) 741 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:29 smithi129 bash[53756]: audit 2023-07-29T17:18:28.620923+0000 mon.a (mon.0) 742 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cephadm 2023-07-29T17:18:27.221001+0000 mgr.y (mgr.25169) 323 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:18:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cluster 2023-07-29T17:18:27.222536+0000 mgr.y (mgr.25169) 324 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:29.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cluster 2023-07-29T17:18:27.222907+0000 mgr.y (mgr.25169) 325 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cephadm 2023-07-29T17:18:27.328391+0000 mgr.y (mgr.25169) 326 : cephadm 1 Upgrade: Setting container_image for all osd 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cephadm 2023-07-29T17:18:27.405956+0000 mgr.y (mgr.25169) 327 : cephadm 1 Upgrade: Setting require_osd_release to 18 reef 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cluster 2023-07-29T17:18:28.398473+0000 mon.a (mon.0) 737 : cluster 1 Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: audit 2023-07-29T17:18:28.407932+0000 mon.a (mon.0) 738 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: cluster 2023-07-29T17:18:28.408057+0000 mon.a (mon.0) 739 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: audit 2023-07-29T17:18:28.412704+0000 mon.a (mon.0) 740 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:29.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: audit 2023-07-29T17:18:28.419700+0000 mon.a (mon.0) 741 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:29.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[58614]: audit 2023-07-29T17:18:28.620923+0000 mon.a (mon.0) 742 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cephadm 2023-07-29T17:18:27.221001+0000 mgr.y (mgr.25169) 323 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:18:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cluster 2023-07-29T17:18:27.222536+0000 mgr.y (mgr.25169) 324 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cluster 2023-07-29T17:18:27.222907+0000 mgr.y (mgr.25169) 325 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cephadm 2023-07-29T17:18:27.328391+0000 mgr.y (mgr.25169) 326 : cephadm 1 Upgrade: Setting container_image for all osd 2023-07-29T17:18:29.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cephadm 2023-07-29T17:18:27.405956+0000 mgr.y (mgr.25169) 327 : cephadm 1 Upgrade: Setting require_osd_release to 18 reef 2023-07-29T17:18:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cluster 2023-07-29T17:18:28.398473+0000 mon.a (mon.0) 737 : cluster 1 Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2023-07-29T17:18:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: audit 2023-07-29T17:18:28.407932+0000 mon.a (mon.0) 738 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2023-07-29T17:18:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: cluster 2023-07-29T17:18:28.408057+0000 mon.a (mon.0) 739 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2023-07-29T17:18:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: audit 2023-07-29T17:18:28.412704+0000 mon.a (mon.0) 740 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: audit 2023-07-29T17:18:28.419700+0000 mon.a (mon.0) 741 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:29.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:29 smithi078 bash[60360]: audit 2023-07-29T17:18:28.620923+0000 mon.a (mon.0) 742 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:30.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:30 smithi078 bash[60360]: cephadm 2023-07-29T17:18:28.414249+0000 mgr.y (mgr.25169) 328 : cephadm 1 Upgrade: Setting container_image for all mds 2023-07-29T17:18:30.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:30 smithi078 bash[58614]: cephadm 2023-07-29T17:18:28.414249+0000 mgr.y (mgr.25169) 328 : cephadm 1 Upgrade: Setting container_image for all mds 2023-07-29T17:18:30.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:30 smithi129 bash[53756]: cephadm 2023-07-29T17:18:28.414249+0000 mgr.y (mgr.25169) 328 : cephadm 1 Upgrade: Setting container_image for all mds 2023-07-29T17:18:31.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:31 smithi129 bash[53756]: cluster 2023-07-29T17:18:29.223443+0000 mgr.y (mgr.25169) 329 : cluster 0 pgmap v236: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:31.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:31 smithi129 bash[53756]: cephadm 2023-07-29T17:18:31.092070+0000 mgr.y (mgr.25169) 330 : cephadm 1 Upgrade: Updating rgw.foo.smithi078.lydpyb (1/4) 2023-07-29T17:18:31.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:31 smithi129 bash[53756]: audit 2023-07-29T17:18:31.099986+0000 mon.a (mon.0) 743 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:31.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:31 smithi129 bash[53756]: audit 2023-07-29T17:18:31.101820+0000 mon.a (mon.0) 744 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:31.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:31 smithi129 bash[53756]: audit 2023-07-29T17:18:31.105599+0000 mon.a (mon.0) 745 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:31.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:31 smithi129 bash[53756]: cephadm 2023-07-29T17:18:31.107212+0000 mgr.y (mgr.25169) 331 : cephadm 1 Deploying daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:18:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[58614]: cluster 2023-07-29T17:18:29.223443+0000 mgr.y (mgr.25169) 329 : cluster 0 pgmap v236: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[58614]: cephadm 2023-07-29T17:18:31.092070+0000 mgr.y (mgr.25169) 330 : cephadm 1 Upgrade: Updating rgw.foo.smithi078.lydpyb (1/4) 2023-07-29T17:18:31.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[58614]: audit 2023-07-29T17:18:31.099986+0000 mon.a (mon.0) 743 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:31.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[58614]: audit 2023-07-29T17:18:31.101820+0000 mon.a (mon.0) 744 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:31.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[58614]: audit 2023-07-29T17:18:31.105599+0000 mon.a (mon.0) 745 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:31.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[58614]: cephadm 2023-07-29T17:18:31.107212+0000 mgr.y (mgr.25169) 331 : cephadm 1 Deploying daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:18:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[60360]: cluster 2023-07-29T17:18:29.223443+0000 mgr.y (mgr.25169) 329 : cluster 0 pgmap v236: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[60360]: cephadm 2023-07-29T17:18:31.092070+0000 mgr.y (mgr.25169) 330 : cephadm 1 Upgrade: Updating rgw.foo.smithi078.lydpyb (1/4) 2023-07-29T17:18:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[60360]: audit 2023-07-29T17:18:31.099986+0000 mon.a (mon.0) 743 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[60360]: audit 2023-07-29T17:18:31.101820+0000 mon.a (mon.0) 744 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.lydpyb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[60360]: audit 2023-07-29T17:18:31.105599+0000 mon.a (mon.0) 745 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:31 smithi078 bash[60360]: cephadm 2023-07-29T17:18:31.107212+0000 mgr.y (mgr.25169) 331 : cephadm 1 Deploying daemon rgw.foo.smithi078.lydpyb on smithi078 2023-07-29T17:18:32.080 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:18:32.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:32 smithi078 bash[58614]: cluster 2023-07-29T17:18:31.224536+0000 mgr.y (mgr.25169) 332 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:32.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:32 smithi078 bash[58614]: audit 2023-07-29T17:18:32.042532+0000 mgr.y (mgr.25169) 333 : audit 0 from='client.34641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:32.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:32 smithi078 bash[60360]: cluster 2023-07-29T17:18:31.224536+0000 mgr.y (mgr.25169) 332 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:32.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:32 smithi078 bash[60360]: audit 2023-07-29T17:18:32.042532+0000 mgr.y (mgr.25169) 333 : audit 0 from='client.34641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:32.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:32 smithi129 bash[53756]: cluster 2023-07-29T17:18:31.224536+0000 mgr.y (mgr.25169) 332 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:32.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:32 smithi129 bash[53756]: audit 2023-07-29T17:18:32.042532+0000 mgr.y (mgr.25169) 333 : audit 0 from='client.34641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:32.831 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:18:32.832 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (8m) 2m ago 14m 10.7M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:18:32.832 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (7m) 16s ago 14m 26.3M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:18:32.832 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 16s ago 12m - - 2023-07-29T17:18:32.832 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (6m) 16s ago 18m 430M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:18:32.832 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (7m) 2m ago 20m 473M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:18:32.833 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6m) 2m ago 20m 64.3M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:18:32.833 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (5m) 16s ago 19m 55.5M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:18:32.833 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5m) 2m ago 19m 48.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:18:32.833 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (8m) 2m ago 14m 8844k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:18:32.833 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (8m) 16s ago 14m 3852k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:18:32.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3m) 2m ago 18m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3m) 2m ago 18m 52.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2m) 2m ago 17m 46.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2m) 2m ago 17m - 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (110s) 16s ago 16m 54.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (80s) 16s ago 16m 49.6M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (51s) 16s ago 16m 49.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (21s) 16s ago 15m - 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:18:32.834 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 16s ago 14m - - 2023-07-29T17:18:32.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (12m) 2m ago 12m 55.8M - 16.2.0 24ecd6d5f14c 9a227b8eb362 2023-07-29T17:18:32.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (12m) 16s ago 12m 56.4M - 16.2.0 24ecd6d5f14c bc0c0ae6f5b2 2023-07-29T17:18:32.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (12m) 2m ago 12m 56.0M - 16.2.0 24ecd6d5f14c 7d6b6bc87c9d 2023-07-29T17:18:32.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (12m) 16s ago 12m 56.3M - 16.2.0 24ecd6d5f14c 4dce8c5d69b5 2023-07-29T17:18:33.271 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:18:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:18:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:18:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:18:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:18:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:18:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4, 2023-07-29T17:18:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 13 2023-07-29T17:18:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:18:33.274 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:18:33.593 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:33 smithi078 bash[60360]: audit 2023-07-29T17:18:32.416921+0000 mgr.y (mgr.25169) 334 : audit 0 from='client.34647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:33.594 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:33 smithi078 bash[60360]: audit 2023-07-29T17:18:32.787134+0000 mgr.y (mgr.25169) 335 : audit 0 from='client.44483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:33.594 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:33 smithi078 bash[58614]: audit 2023-07-29T17:18:32.416921+0000 mgr.y (mgr.25169) 334 : audit 0 from='client.34647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:33.594 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:33 smithi078 bash[58614]: audit 2023-07-29T17:18:32.787134+0000 mgr.y (mgr.25169) 335 : audit 0 from='client.44483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:33.671 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:18:33.672 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:18:33.672 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: "osd", 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:18:33.673 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "13/23 daemons upgraded", 2023-07-29T17:18:33.674 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2023-07-29T17:18:33.674 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:18:33.674 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:18:33.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:33 smithi129 bash[53756]: audit 2023-07-29T17:18:32.416921+0000 mgr.y (mgr.25169) 334 : audit 0 from='client.34647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:33.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:33 smithi129 bash[53756]: audit 2023-07-29T17:18:32.787134+0000 mgr.y (mgr.25169) 335 : audit 0 from='client.44483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:18:34.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[58614]: cluster 2023-07-29T17:18:33.225823+0000 mgr.y (mgr.25169) 336 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:34.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[58614]: audit 2023-07-29T17:18:33.248779+0000 mon.a (mon.0) 746 : audit 0 from='client.34659 172.21.15.78:0/1208068945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:34.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[58614]: audit 2023-07-29T17:18:33.648280+0000 mgr.y (mgr.25169) 337 : audit 0 from='client.34665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:34.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[58614]: audit 2023-07-29T17:18:34.089931+0000 mon.c (mon.2) 13 : audit 0 from='client.34671 172.21.15.78:0/3373805085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:18:34.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[60360]: cluster 2023-07-29T17:18:33.225823+0000 mgr.y (mgr.25169) 336 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:34.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[60360]: audit 2023-07-29T17:18:33.248779+0000 mon.a (mon.0) 746 : audit 0 from='client.34659 172.21.15.78:0/1208068945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:34.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[60360]: audit 2023-07-29T17:18:33.648280+0000 mgr.y (mgr.25169) 337 : audit 0 from='client.34665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:34.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:34 smithi078 bash[60360]: audit 2023-07-29T17:18:34.089931+0000 mon.c (mon.2) 13 : audit 0 from='client.34671 172.21.15.78:0/3373805085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:18:34.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:34 smithi129 bash[53756]: cluster 2023-07-29T17:18:33.225823+0000 mgr.y (mgr.25169) 336 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:34.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:34 smithi129 bash[53756]: audit 2023-07-29T17:18:33.248779+0000 mon.a (mon.0) 746 : audit 0 from='client.34659 172.21.15.78:0/1208068945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:18:34.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:34 smithi129 bash[53756]: audit 2023-07-29T17:18:33.648280+0000 mgr.y (mgr.25169) 337 : audit 0 from='client.34665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:18:34.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:34 smithi129 bash[53756]: audit 2023-07-29T17:18:34.089931+0000 mon.c (mon.2) 13 : audit 0 from='client.34671 172.21.15.78:0/3373805085' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:18:36.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:36 smithi129 bash[53756]: cluster 2023-07-29T17:18:35.226536+0000 mgr.y (mgr.25169) 338 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:36.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:36 smithi129 bash[53756]: audit 2023-07-29T17:18:35.275822+0000 mon.a (mon.0) 747 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:36.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:36 smithi129 bash[53756]: audit 2023-07-29T17:18:35.284279+0000 mon.a (mon.0) 748 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:36.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:36 smithi078 bash[58614]: cluster 2023-07-29T17:18:35.226536+0000 mgr.y (mgr.25169) 338 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:36.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:36 smithi078 bash[58614]: audit 2023-07-29T17:18:35.275822+0000 mon.a (mon.0) 747 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:36.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:36 smithi078 bash[58614]: audit 2023-07-29T17:18:35.284279+0000 mon.a (mon.0) 748 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:36.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:36 smithi078 bash[60360]: cluster 2023-07-29T17:18:35.226536+0000 mgr.y (mgr.25169) 338 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:18:36.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:36 smithi078 bash[60360]: audit 2023-07-29T17:18:35.275822+0000 mon.a (mon.0) 747 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:36.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:36 smithi078 bash[60360]: audit 2023-07-29T17:18:35.284279+0000 mon.a (mon.0) 748 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:39.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[60360]: cluster 2023-07-29T17:18:37.227332+0000 mgr.y (mgr.25169) 339 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:18:39.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[60360]: cephadm 2023-07-29T17:18:38.154517+0000 mgr.y (mgr.25169) 340 : cephadm 1 Upgrade: Updating rgw.smpl.smithi078.xbxtjr (2/4) 2023-07-29T17:18:39.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[60360]: audit 2023-07-29T17:18:38.164755+0000 mon.a (mon.0) 749 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:39.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[60360]: audit 2023-07-29T17:18:38.167028+0000 mon.a (mon.0) 750 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:39.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[60360]: audit 2023-07-29T17:18:38.170824+0000 mon.a (mon.0) 751 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:39.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[60360]: cephadm 2023-07-29T17:18:38.172519+0000 mgr.y (mgr.25169) 341 : cephadm 1 Deploying daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:18:39.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[58614]: cluster 2023-07-29T17:18:37.227332+0000 mgr.y (mgr.25169) 339 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:18:39.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[58614]: cephadm 2023-07-29T17:18:38.154517+0000 mgr.y (mgr.25169) 340 : cephadm 1 Upgrade: Updating rgw.smpl.smithi078.xbxtjr (2/4) 2023-07-29T17:18:39.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[58614]: audit 2023-07-29T17:18:38.164755+0000 mon.a (mon.0) 749 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:39.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[58614]: audit 2023-07-29T17:18:38.167028+0000 mon.a (mon.0) 750 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:39.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[58614]: audit 2023-07-29T17:18:38.170824+0000 mon.a (mon.0) 751 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:39.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:39 smithi078 bash[58614]: cephadm 2023-07-29T17:18:38.172519+0000 mgr.y (mgr.25169) 341 : cephadm 1 Deploying daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:18:39.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:39 smithi129 bash[53756]: cluster 2023-07-29T17:18:37.227332+0000 mgr.y (mgr.25169) 339 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-07-29T17:18:39.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:39 smithi129 bash[53756]: cephadm 2023-07-29T17:18:38.154517+0000 mgr.y (mgr.25169) 340 : cephadm 1 Upgrade: Updating rgw.smpl.smithi078.xbxtjr (2/4) 2023-07-29T17:18:39.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:39 smithi129 bash[53756]: audit 2023-07-29T17:18:38.164755+0000 mon.a (mon.0) 749 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:39.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:39 smithi129 bash[53756]: audit 2023-07-29T17:18:38.167028+0000 mon.a (mon.0) 750 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.xbxtjr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:39.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:39 smithi129 bash[53756]: audit 2023-07-29T17:18:38.170824+0000 mon.a (mon.0) 751 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:39.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:39 smithi129 bash[53756]: cephadm 2023-07-29T17:18:38.172519+0000 mgr.y (mgr.25169) 341 : cephadm 1 Deploying daemon rgw.smpl.smithi078.xbxtjr on smithi078 2023-07-29T17:18:41.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:41 smithi078 bash[58614]: cluster 2023-07-29T17:18:39.228648+0000 mgr.y (mgr.25169) 342 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2023-07-29T17:18:41.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:41 smithi078 bash[60360]: cluster 2023-07-29T17:18:39.228648+0000 mgr.y (mgr.25169) 342 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2023-07-29T17:18:41.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:41 smithi129 bash[53756]: cluster 2023-07-29T17:18:39.228648+0000 mgr.y (mgr.25169) 342 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2023-07-29T17:18:43.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:43 smithi078 bash[58614]: cluster 2023-07-29T17:18:41.230169+0000 mgr.y (mgr.25169) 343 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 300 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 170 B/s wr, 60 op/s 2023-07-29T17:18:43.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:43 smithi078 bash[58614]: audit 2023-07-29T17:18:42.210359+0000 mon.a (mon.0) 752 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:43.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:43 smithi078 bash[58614]: audit 2023-07-29T17:18:42.217014+0000 mon.a (mon.0) 753 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:43.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:43 smithi078 bash[60360]: cluster 2023-07-29T17:18:41.230169+0000 mgr.y (mgr.25169) 343 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 300 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 170 B/s wr, 60 op/s 2023-07-29T17:18:43.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:43 smithi078 bash[60360]: audit 2023-07-29T17:18:42.210359+0000 mon.a (mon.0) 752 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:43.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:43 smithi078 bash[60360]: audit 2023-07-29T17:18:42.217014+0000 mon.a (mon.0) 753 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:43.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:43 smithi129 bash[53756]: cluster 2023-07-29T17:18:41.230169+0000 mgr.y (mgr.25169) 343 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 300 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 170 B/s wr, 60 op/s 2023-07-29T17:18:43.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:43 smithi129 bash[53756]: audit 2023-07-29T17:18:42.210359+0000 mon.a (mon.0) 752 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:43.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:43 smithi129 bash[53756]: audit 2023-07-29T17:18:42.217014+0000 mon.a (mon.0) 753 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:45.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:45 smithi129 bash[53756]: cluster 2023-07-29T17:18:43.231170+0000 mgr.y (mgr.25169) 344 : cluster 0 pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 300 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 170 B/s wr, 103 op/s 2023-07-29T17:18:45.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:45 smithi129 bash[53756]: audit 2023-07-29T17:18:44.925226+0000 mon.a (mon.0) 754 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:45.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:45 smithi129 bash[53756]: audit 2023-07-29T17:18:44.927227+0000 mon.a (mon.0) 755 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:45.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:45 smithi129 bash[53756]: audit 2023-07-29T17:18:44.930671+0000 mon.a (mon.0) 756 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:45.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[58614]: cluster 2023-07-29T17:18:43.231170+0000 mgr.y (mgr.25169) 344 : cluster 0 pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 300 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 170 B/s wr, 103 op/s 2023-07-29T17:18:45.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[58614]: audit 2023-07-29T17:18:44.925226+0000 mon.a (mon.0) 754 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:45.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[58614]: audit 2023-07-29T17:18:44.927227+0000 mon.a (mon.0) 755 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:45.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[58614]: audit 2023-07-29T17:18:44.930671+0000 mon.a (mon.0) 756 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:45.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[60360]: cluster 2023-07-29T17:18:43.231170+0000 mgr.y (mgr.25169) 344 : cluster 0 pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 300 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 170 B/s wr, 103 op/s 2023-07-29T17:18:45.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[60360]: audit 2023-07-29T17:18:44.925226+0000 mon.a (mon.0) 754 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:45.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[60360]: audit 2023-07-29T17:18:44.927227+0000 mon.a (mon.0) 755 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.dbexjf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:45.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:45 smithi078 bash[60360]: audit 2023-07-29T17:18:44.930671+0000 mon.a (mon.0) 756 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:46.539 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:46 smithi129 bash[53756]: cephadm 2023-07-29T17:18:44.915730+0000 mgr.y (mgr.25169) 345 : cephadm 1 Upgrade: Updating rgw.foo.smithi129.dbexjf (3/4) 2023-07-29T17:18:46.539 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:46 smithi129 bash[53756]: cephadm 2023-07-29T17:18:44.932548+0000 mgr.y (mgr.25169) 346 : cephadm 1 Deploying daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:18:46.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:46 smithi078 bash[58614]: cephadm 2023-07-29T17:18:44.915730+0000 mgr.y (mgr.25169) 345 : cephadm 1 Upgrade: Updating rgw.foo.smithi129.dbexjf (3/4) 2023-07-29T17:18:46.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:46 smithi078 bash[58614]: cephadm 2023-07-29T17:18:44.932548+0000 mgr.y (mgr.25169) 346 : cephadm 1 Deploying daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:18:46.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:46 smithi078 bash[60360]: cephadm 2023-07-29T17:18:44.915730+0000 mgr.y (mgr.25169) 345 : cephadm 1 Upgrade: Updating rgw.foo.smithi129.dbexjf (3/4) 2023-07-29T17:18:46.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:46 smithi078 bash[60360]: cephadm 2023-07-29T17:18:44.932548+0000 mgr.y (mgr.25169) 346 : cephadm 1 Deploying daemon rgw.foo.smithi129.dbexjf on smithi129 2023-07-29T17:18:47.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:47 smithi129 bash[53756]: cluster 2023-07-29T17:18:45.232491+0000 mgr.y (mgr.25169) 347 : cluster 0 pgmap v244: 161 pgs: 161 active+clean; 604 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 255 B/s wr, 128 op/s 2023-07-29T17:18:47.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:47 smithi078 bash[58614]: cluster 2023-07-29T17:18:45.232491+0000 mgr.y (mgr.25169) 347 : cluster 0 pgmap v244: 161 pgs: 161 active+clean; 604 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 255 B/s wr, 128 op/s 2023-07-29T17:18:47.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:47 smithi078 bash[60360]: cluster 2023-07-29T17:18:45.232491+0000 mgr.y (mgr.25169) 347 : cluster 0 pgmap v244: 161 pgs: 161 active+clean; 604 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 255 B/s wr, 128 op/s 2023-07-29T17:18:48.557 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:48 smithi129 bash[53756]: cluster 2023-07-29T17:18:47.233706+0000 mgr.y (mgr.25169) 348 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 255 B/s wr, 162 op/s 2023-07-29T17:18:48.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:48 smithi078 bash[58614]: cluster 2023-07-29T17:18:47.233706+0000 mgr.y (mgr.25169) 348 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 255 B/s wr, 162 op/s 2023-07-29T17:18:48.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:48 smithi078 bash[60360]: cluster 2023-07-29T17:18:47.233706+0000 mgr.y (mgr.25169) 348 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 255 B/s wr, 162 op/s 2023-07-29T17:18:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:49 smithi129 bash[53756]: audit 2023-07-29T17:18:48.828897+0000 mon.a (mon.0) 757 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:50.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:49 smithi129 bash[53756]: audit 2023-07-29T17:18:48.842270+0000 mon.a (mon.0) 758 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:49 smithi078 bash[58614]: audit 2023-07-29T17:18:48.828897+0000 mon.a (mon.0) 757 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:50.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:49 smithi078 bash[58614]: audit 2023-07-29T17:18:48.842270+0000 mon.a (mon.0) 758 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:49 smithi078 bash[60360]: audit 2023-07-29T17:18:48.828897+0000 mon.a (mon.0) 757 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:50.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:49 smithi078 bash[60360]: audit 2023-07-29T17:18:48.842270+0000 mon.a (mon.0) 758 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:51.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:50 smithi129 bash[53756]: cluster 2023-07-29T17:18:49.235114+0000 mgr.y (mgr.25169) 349 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2023-07-29T17:18:51.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:50 smithi078 bash[60360]: cluster 2023-07-29T17:18:49.235114+0000 mgr.y (mgr.25169) 349 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2023-07-29T17:18:51.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:50 smithi078 bash[58614]: cluster 2023-07-29T17:18:49.235114+0000 mgr.y (mgr.25169) 349 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2023-07-29T17:18:52.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:52 smithi129 bash[53756]: cluster 2023-07-29T17:18:51.236215+0000 mgr.y (mgr.25169) 350 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 255 B/s wr, 156 op/s 2023-07-29T17:18:52.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:52 smithi129 bash[53756]: cephadm 2023-07-29T17:18:51.477480+0000 mgr.y (mgr.25169) 351 : cephadm 1 Upgrade: Updating rgw.smpl.smithi129.tqsnpp (4/4) 2023-07-29T17:18:52.847 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:52 smithi129 bash[53756]: audit 2023-07-29T17:18:51.486660+0000 mon.a (mon.0) 759 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:52.847 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:52 smithi129 bash[53756]: audit 2023-07-29T17:18:51.488358+0000 mon.a (mon.0) 760 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:52.847 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:52 smithi129 bash[53756]: audit 2023-07-29T17:18:51.491002+0000 mon.a (mon.0) 761 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:52.847 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:52 smithi129 bash[53756]: cephadm 2023-07-29T17:18:51.492435+0000 mgr.y (mgr.25169) 352 : cephadm 1 Deploying daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:18:52.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[58614]: cluster 2023-07-29T17:18:51.236215+0000 mgr.y (mgr.25169) 350 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 255 B/s wr, 156 op/s 2023-07-29T17:18:52.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[58614]: cephadm 2023-07-29T17:18:51.477480+0000 mgr.y (mgr.25169) 351 : cephadm 1 Upgrade: Updating rgw.smpl.smithi129.tqsnpp (4/4) 2023-07-29T17:18:52.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[58614]: audit 2023-07-29T17:18:51.486660+0000 mon.a (mon.0) 759 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:52.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[58614]: audit 2023-07-29T17:18:51.488358+0000 mon.a (mon.0) 760 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:52.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[58614]: audit 2023-07-29T17:18:51.491002+0000 mon.a (mon.0) 761 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:52.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[58614]: cephadm 2023-07-29T17:18:51.492435+0000 mgr.y (mgr.25169) 352 : cephadm 1 Deploying daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:18:52.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[60360]: cluster 2023-07-29T17:18:51.236215+0000 mgr.y (mgr.25169) 350 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 255 B/s wr, 156 op/s 2023-07-29T17:18:52.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[60360]: cephadm 2023-07-29T17:18:51.477480+0000 mgr.y (mgr.25169) 351 : cephadm 1 Upgrade: Updating rgw.smpl.smithi129.tqsnpp (4/4) 2023-07-29T17:18:52.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[60360]: audit 2023-07-29T17:18:51.486660+0000 mon.a (mon.0) 759 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:52.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[60360]: audit 2023-07-29T17:18:51.488358+0000 mon.a (mon.0) 760 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.tqsnpp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-29T17:18:52.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[60360]: audit 2023-07-29T17:18:51.491002+0000 mon.a (mon.0) 761 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:18:52.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:52 smithi078 bash[60360]: cephadm 2023-07-29T17:18:51.492435+0000 mgr.y (mgr.25169) 352 : cephadm 1 Deploying daemon rgw.smpl.smithi129.tqsnpp on smithi129 2023-07-29T17:18:55.161 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:54 smithi129 bash[53756]: cluster 2023-07-29T17:18:53.237588+0000 mgr.y (mgr.25169) 353 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 85 B/s wr, 169 op/s 2023-07-29T17:18:55.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:54 smithi078 bash[58614]: cluster 2023-07-29T17:18:53.237588+0000 mgr.y (mgr.25169) 353 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 85 B/s wr, 169 op/s 2023-07-29T17:18:55.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:54 smithi078 bash[60360]: cluster 2023-07-29T17:18:53.237588+0000 mgr.y (mgr.25169) 353 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 85 B/s wr, 169 op/s 2023-07-29T17:18:56.901 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:56 smithi129 bash[53756]: cluster 2023-07-29T17:18:55.238893+0000 mgr.y (mgr.25169) 354 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 604 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-29T17:18:56.901 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:56 smithi129 bash[53756]: audit 2023-07-29T17:18:55.551751+0000 mon.a (mon.0) 762 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.902 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:56 smithi129 bash[53756]: audit 2023-07-29T17:18:55.559902+0000 mon.a (mon.0) 763 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.902 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:56 smithi129 bash[53756]: audit 2023-07-29T17:18:55.942948+0000 mon.a (mon.0) 764 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.902 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:56 smithi129 bash[53756]: audit 2023-07-29T17:18:55.953098+0000 mon.a (mon.0) 765 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[60360]: cluster 2023-07-29T17:18:55.238893+0000 mgr.y (mgr.25169) 354 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 604 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-29T17:18:56.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[60360]: audit 2023-07-29T17:18:55.551751+0000 mon.a (mon.0) 762 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[60360]: audit 2023-07-29T17:18:55.559902+0000 mon.a (mon.0) 763 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[60360]: audit 2023-07-29T17:18:55.942948+0000 mon.a (mon.0) 764 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[60360]: audit 2023-07-29T17:18:55.953098+0000 mon.a (mon.0) 765 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[58614]: cluster 2023-07-29T17:18:55.238893+0000 mgr.y (mgr.25169) 354 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 604 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-29T17:18:56.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[58614]: audit 2023-07-29T17:18:55.551751+0000 mon.a (mon.0) 762 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[58614]: audit 2023-07-29T17:18:55.559902+0000 mon.a (mon.0) 763 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[58614]: audit 2023-07-29T17:18:55.942948+0000 mon.a (mon.0) 764 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:56.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:56 smithi078 bash[58614]: audit 2023-07-29T17:18:55.953098+0000 mon.a (mon.0) 765 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:18:58.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:18:58 smithi129 bash[53756]: cluster 2023-07-29T17:18:57.240429+0000 mgr.y (mgr.25169) 355 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 85 B/s wr, 150 op/s 2023-07-29T17:18:58.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:18:58 smithi078 bash[58614]: cluster 2023-07-29T17:18:57.240429+0000 mgr.y (mgr.25169) 355 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 85 B/s wr, 150 op/s 2023-07-29T17:18:58.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:18:58 smithi078 bash[60360]: cluster 2023-07-29T17:18:57.240429+0000 mgr.y (mgr.25169) 355 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 85 B/s wr, 150 op/s 2023-07-29T17:19:01.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:00 smithi129 bash[53756]: cluster 2023-07-29T17:18:59.241839+0000 mgr.y (mgr.25169) 356 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 154 op/s 2023-07-29T17:19:01.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:00 smithi078 bash[60360]: cluster 2023-07-29T17:18:59.241839+0000 mgr.y (mgr.25169) 356 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 154 op/s 2023-07-29T17:19:01.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:00 smithi078 bash[58614]: cluster 2023-07-29T17:18:59.241839+0000 mgr.y (mgr.25169) 356 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 154 op/s 2023-07-29T17:19:03.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: cluster 2023-07-29T17:19:01.242854+0000 mgr.y (mgr.25169) 357 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2023-07-29T17:19:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: audit 2023-07-29T17:19:01.758660+0000 mon.a (mon.0) 766 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: audit 2023-07-29T17:19:01.768820+0000 mon.a (mon.0) 767 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: audit 2023-07-29T17:19:02.055385+0000 mon.a (mon.0) 768 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: audit 2023-07-29T17:19:02.065344+0000 mon.a (mon.0) 769 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: audit 2023-07-29T17:19:02.577650+0000 mon.a (mon.0) 770 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:02 smithi129 bash[53756]: audit 2023-07-29T17:19:02.586787+0000 mon.a (mon.0) 771 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: cluster 2023-07-29T17:19:01.242854+0000 mgr.y (mgr.25169) 357 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2023-07-29T17:19:03.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: audit 2023-07-29T17:19:01.758660+0000 mon.a (mon.0) 766 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: audit 2023-07-29T17:19:01.768820+0000 mon.a (mon.0) 767 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: audit 2023-07-29T17:19:02.055385+0000 mon.a (mon.0) 768 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: audit 2023-07-29T17:19:02.065344+0000 mon.a (mon.0) 769 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: audit 2023-07-29T17:19:02.577650+0000 mon.a (mon.0) 770 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[58614]: audit 2023-07-29T17:19:02.586787+0000 mon.a (mon.0) 771 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: cluster 2023-07-29T17:19:01.242854+0000 mgr.y (mgr.25169) 357 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2023-07-29T17:19:03.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: audit 2023-07-29T17:19:01.758660+0000 mon.a (mon.0) 766 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: audit 2023-07-29T17:19:01.768820+0000 mon.a (mon.0) 767 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: audit 2023-07-29T17:19:02.055385+0000 mon.a (mon.0) 768 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: audit 2023-07-29T17:19:02.065344+0000 mon.a (mon.0) 769 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: audit 2023-07-29T17:19:02.577650+0000 mon.a (mon.0) 770 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:02 smithi078 bash[60360]: audit 2023-07-29T17:19:02.586787+0000 mon.a (mon.0) 771 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:03 smithi129 bash[53756]: audit 2023-07-29T17:19:02.872373+0000 mon.a (mon.0) 772 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:03 smithi129 bash[53756]: audit 2023-07-29T17:19:02.883516+0000 mon.a (mon.0) 773 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:03 smithi078 bash[58614]: audit 2023-07-29T17:19:02.872373+0000 mon.a (mon.0) 772 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:03 smithi078 bash[58614]: audit 2023-07-29T17:19:02.883516+0000 mon.a (mon.0) 773 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:03 smithi078 bash[60360]: audit 2023-07-29T17:19:02.872373+0000 mon.a (mon.0) 772 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:03 smithi078 bash[60360]: audit 2023-07-29T17:19:02.883516+0000 mon.a (mon.0) 773 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:04.515 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:19:05.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:04 smithi129 bash[53756]: cluster 2023-07-29T17:19:03.244305+0000 mgr.y (mgr.25169) 358 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-29T17:19:05.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:04 smithi078 bash[60360]: cluster 2023-07-29T17:19:03.244305+0000 mgr.y (mgr.25169) 358 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-29T17:19:05.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:04 smithi078 bash[58614]: cluster 2023-07-29T17:19:03.244305+0000 mgr.y (mgr.25169) 358 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-29T17:19:05.263 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:19:05.263 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (9m) 3s ago 15m 10.7M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:19:05.264 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (8m) 3s ago 14m 26.4M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:19:05.264 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 error 3s ago 12m - - 2023-07-29T17:19:05.264 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (7m) 3s ago 19m 430M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:19:05.264 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (8m) 3s ago 21m 477M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:19:05.264 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6m) 3s ago 21m 78.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:19:05.264 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (5m) 3s ago 19m 57.4M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6m) 3s ago 19m 58.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (9m) 3s ago 15m 5940k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (8m) 3s ago 15m 3888k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4m) 3s ago 19m 61.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3m) 3s ago 18m 63.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3m) 3s ago 18m 52.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:19:05.265 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2m) 3s ago 17m 59.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (2m) 3s ago 17m 60.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (113s) 3s ago 17m 53.7M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (83s) 3s ago 16m 53.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (54s) 3s ago 16m 54.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 3s ago 15m - - 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (29s) 3s ago 13m 83.5M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (16s) 3s ago 13m 83.5M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:19:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (22s) 3s ago 13m 83.6M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:19:05.267 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (9s) 3s ago 13m 83.4M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:19:05.685 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:19:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:19:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:19:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:19:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:19:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:19:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:19:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:19:05.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:19:06.063 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:19:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:19:06.063 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:19:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:19:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:19:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2023-07-29T17:19:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "osd", 2023-07-29T17:19:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:19:06.064 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:19:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:19:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2023-07-29T17:19:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2023-07-29T17:19:06.065 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:19:06.066 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:19:06.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:05 smithi129 bash[53756]: audit 2023-07-29T17:19:04.476993+0000 mgr.y (mgr.25169) 359 : audit 0 from='client.44591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:06.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:05 smithi129 bash[53756]: audit 2023-07-29T17:19:04.852593+0000 mgr.y (mgr.25169) 360 : audit 0 from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:06.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:05 smithi129 bash[53756]: audit 2023-07-29T17:19:05.662401+0000 mon.c (mon.2) 14 : audit 0 from='client.44609 172.21.15.78:0/2747949948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:06.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:05 smithi078 bash[60360]: audit 2023-07-29T17:19:04.476993+0000 mgr.y (mgr.25169) 359 : audit 0 from='client.44591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:06.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:05 smithi078 bash[60360]: audit 2023-07-29T17:19:04.852593+0000 mgr.y (mgr.25169) 360 : audit 0 from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:06.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:05 smithi078 bash[60360]: audit 2023-07-29T17:19:05.662401+0000 mon.c (mon.2) 14 : audit 0 from='client.44609 172.21.15.78:0/2747949948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:06.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:05 smithi078 bash[58614]: audit 2023-07-29T17:19:04.476993+0000 mgr.y (mgr.25169) 359 : audit 0 from='client.44591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:06.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:05 smithi078 bash[58614]: audit 2023-07-29T17:19:04.852593+0000 mgr.y (mgr.25169) 360 : audit 0 from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:06.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:05 smithi078 bash[58614]: audit 2023-07-29T17:19:05.662401+0000 mon.c (mon.2) 14 : audit 0 from='client.44609 172.21.15.78:0/2747949948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:06.498 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-29T17:19:06.498 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-29T17:19:06.499 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:19:06.499 INFO:teuthology.orchestra.run.smithi078.stdout: daemon iscsi.foo.smithi129.vbvpri on smithi129 is in error state 2023-07-29T17:19:07.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:06 smithi129 bash[53756]: audit 2023-07-29T17:19:05.221567+0000 mgr.y (mgr.25169) 361 : audit 0 from='client.44603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:06 smithi129 bash[53756]: cluster 2023-07-29T17:19:05.245447+0000 mgr.y (mgr.25169) 362 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 170 B/s wr, 121 op/s 2023-07-29T17:19:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:06 smithi129 bash[53756]: audit 2023-07-29T17:19:06.039360+0000 mgr.y (mgr.25169) 363 : audit 0 from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:06 smithi129 bash[53756]: audit 2023-07-29T17:19:06.475246+0000 mon.a (mon.0) 774 : audit 0 from='client.34788 172.21.15.78:0/2905062038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:19:07.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[60360]: audit 2023-07-29T17:19:05.221567+0000 mgr.y (mgr.25169) 361 : audit 0 from='client.44603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:07.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[60360]: cluster 2023-07-29T17:19:05.245447+0000 mgr.y (mgr.25169) 362 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 170 B/s wr, 121 op/s 2023-07-29T17:19:07.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[60360]: audit 2023-07-29T17:19:06.039360+0000 mgr.y (mgr.25169) 363 : audit 0 from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:07.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[60360]: audit 2023-07-29T17:19:06.475246+0000 mon.a (mon.0) 774 : audit 0 from='client.34788 172.21.15.78:0/2905062038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:19:07.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[58614]: audit 2023-07-29T17:19:05.221567+0000 mgr.y (mgr.25169) 361 : audit 0 from='client.44603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:07.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[58614]: cluster 2023-07-29T17:19:05.245447+0000 mgr.y (mgr.25169) 362 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 170 B/s wr, 121 op/s 2023-07-29T17:19:07.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[58614]: audit 2023-07-29T17:19:06.039360+0000 mgr.y (mgr.25169) 363 : audit 0 from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:07.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:06 smithi078 bash[58614]: audit 2023-07-29T17:19:06.475246+0000 mon.a (mon.0) 774 : audit 0 from='client.34788 172.21.15.78:0/2905062038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:19:09.094 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:08 smithi129 bash[53756]: cluster 2023-07-29T17:19:07.246724+0000 mgr.y (mgr.25169) 364 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 85 B/s wr, 82 op/s 2023-07-29T17:19:09.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:08 smithi078 bash[58614]: cluster 2023-07-29T17:19:07.246724+0000 mgr.y (mgr.25169) 364 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 85 B/s wr, 82 op/s 2023-07-29T17:19:09.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:08 smithi078 bash[60360]: cluster 2023-07-29T17:19:07.246724+0000 mgr.y (mgr.25169) 364 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 85 B/s wr, 82 op/s 2023-07-29T17:19:11.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:10 smithi078 bash[58614]: cluster 2023-07-29T17:19:09.248020+0000 mgr.y (mgr.25169) 365 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 73 op/s 2023-07-29T17:19:11.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:10 smithi078 bash[60360]: cluster 2023-07-29T17:19:09.248020+0000 mgr.y (mgr.25169) 365 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 73 op/s 2023-07-29T17:19:11.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:10 smithi129 bash[53756]: cluster 2023-07-29T17:19:09.248020+0000 mgr.y (mgr.25169) 365 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 73 op/s 2023-07-29T17:19:13.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: cluster 2023-07-29T17:19:11.248981+0000 mgr.y (mgr.25169) 366 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 35 op/s 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: cephadm 2023-07-29T17:19:12.175186+0000 mgr.y (mgr.25169) 367 : cephadm 1 Detected new or changed devices on smithi129 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.187795+0000 mon.a (mon.0) 775 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.200172+0000 mon.a (mon.0) 776 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.635812+0000 mon.a (mon.0) 777 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.648940+0000 mon.a (mon.0) 778 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.650734+0000 mon.a (mon.0) 779 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.652437+0000 mon.a (mon.0) 780 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.663113+0000 mon.a (mon.0) 781 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.756069+0000 mon.a (mon.0) 782 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.761327+0000 mon.a (mon.0) 783 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.764323+0000 mon.a (mon.0) 784 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.766624+0000 mon.a (mon.0) 785 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.770248+0000 mon.a (mon.0) 786 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.774099+0000 mon.a (mon.0) 787 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.777087+0000 mon.a (mon.0) 788 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.790269+0000 mon.a (mon.0) 789 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.792194+0000 mon.a (mon.0) 790 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi078.lydpyb"}]: dispatch 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.801290+0000 mon.a (mon.0) 791 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi078.lydpyb"}]': finished 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.802615+0000 mon.a (mon.0) 792 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.dbexjf"}]: dispatch 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.810507+0000 mon.a (mon.0) 793 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.dbexjf"}]': finished 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.812238+0000 mon.a (mon.0) 794 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi078.xbxtjr"}]: dispatch 2023-07-29T17:19:13.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.821240+0000 mon.a (mon.0) 795 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi078.xbxtjr"}]': finished 2023-07-29T17:19:13.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.822755+0000 mon.a (mon.0) 796 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.tqsnpp"}]: dispatch 2023-07-29T17:19:13.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.832040+0000 mon.a (mon.0) 797 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.tqsnpp"}]': finished 2023-07-29T17:19:13.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.834186+0000 mon.a (mon.0) 798 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.844103+0000 mon.a (mon.0) 799 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.846347+0000 mon.a (mon.0) 800 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.680 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.848679+0000 mon.a (mon.0) 801 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.681 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:13 smithi129 bash[53756]: audit 2023-07-29T17:19:12.856734+0000 mon.a (mon.0) 802 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: cluster 2023-07-29T17:19:11.248981+0000 mgr.y (mgr.25169) 366 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 35 op/s 2023-07-29T17:19:13.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: cephadm 2023-07-29T17:19:12.175186+0000 mgr.y (mgr.25169) 367 : cephadm 1 Detected new or changed devices on smithi129 2023-07-29T17:19:13.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.187795+0000 mon.a (mon.0) 775 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.200172+0000 mon.a (mon.0) 776 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.635812+0000 mon.a (mon.0) 777 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.648940+0000 mon.a (mon.0) 778 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.650734+0000 mon.a (mon.0) 779 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.652437+0000 mon.a (mon.0) 780 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.663113+0000 mon.a (mon.0) 781 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.756069+0000 mon.a (mon.0) 782 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.761327+0000 mon.a (mon.0) 783 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.764323+0000 mon.a (mon.0) 784 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.766624+0000 mon.a (mon.0) 785 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.770248+0000 mon.a (mon.0) 786 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.774099+0000 mon.a (mon.0) 787 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.777087+0000 mon.a (mon.0) 788 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.790269+0000 mon.a (mon.0) 789 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.792194+0000 mon.a (mon.0) 790 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi078.lydpyb"}]: dispatch 2023-07-29T17:19:13.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.801290+0000 mon.a (mon.0) 791 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi078.lydpyb"}]': finished 2023-07-29T17:19:13.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.802615+0000 mon.a (mon.0) 792 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.dbexjf"}]: dispatch 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: cluster 2023-07-29T17:19:11.248981+0000 mgr.y (mgr.25169) 366 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 35 op/s 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: cephadm 2023-07-29T17:19:12.175186+0000 mgr.y (mgr.25169) 367 : cephadm 1 Detected new or changed devices on smithi129 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.187795+0000 mon.a (mon.0) 775 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.200172+0000 mon.a (mon.0) 776 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.635812+0000 mon.a (mon.0) 777 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.648940+0000 mon.a (mon.0) 778 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.650734+0000 mon.a (mon.0) 779 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.652437+0000 mon.a (mon.0) 780 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.663113+0000 mon.a (mon.0) 781 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.756069+0000 mon.a (mon.0) 782 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.761327+0000 mon.a (mon.0) 783 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.764323+0000 mon.a (mon.0) 784 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.766624+0000 mon.a (mon.0) 785 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.770248+0000 mon.a (mon.0) 786 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.774099+0000 mon.a (mon.0) 787 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.777087+0000 mon.a (mon.0) 788 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.790269+0000 mon.a (mon.0) 789 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.792194+0000 mon.a (mon.0) 790 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi078.lydpyb"}]: dispatch 2023-07-29T17:19:13.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.801290+0000 mon.a (mon.0) 791 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi078.lydpyb"}]': finished 2023-07-29T17:19:13.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.802615+0000 mon.a (mon.0) 792 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.dbexjf"}]: dispatch 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.810507+0000 mon.a (mon.0) 793 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.dbexjf"}]': finished 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.812238+0000 mon.a (mon.0) 794 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi078.xbxtjr"}]: dispatch 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.821240+0000 mon.a (mon.0) 795 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi078.xbxtjr"}]': finished 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.822755+0000 mon.a (mon.0) 796 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.tqsnpp"}]: dispatch 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.832040+0000 mon.a (mon.0) 797 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.tqsnpp"}]': finished 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.834186+0000 mon.a (mon.0) 798 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.844103+0000 mon.a (mon.0) 799 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.846347+0000 mon.a (mon.0) 800 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.848679+0000 mon.a (mon.0) 801 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[58614]: audit 2023-07-29T17:19:12.856734+0000 mon.a (mon.0) 802 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.810507+0000 mon.a (mon.0) 793 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.dbexjf"}]': finished 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.812238+0000 mon.a (mon.0) 794 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi078.xbxtjr"}]: dispatch 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.821240+0000 mon.a (mon.0) 795 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi078.xbxtjr"}]': finished 2023-07-29T17:19:13.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.822755+0000 mon.a (mon.0) 796 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.tqsnpp"}]: dispatch 2023-07-29T17:19:13.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.832040+0000 mon.a (mon.0) 797 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.tqsnpp"}]': finished 2023-07-29T17:19:13.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.834186+0000 mon.a (mon.0) 798 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.844103+0000 mon.a (mon.0) 799 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:13.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.846347+0000 mon.a (mon.0) 800 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.848679+0000 mon.a (mon.0) 801 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:13.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:13 smithi078 bash[60360]: audit 2023-07-29T17:19:12.856734+0000 mon.a (mon.0) 802 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:14.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cephadm 2023-07-29T17:19:12.623237+0000 mgr.y (mgr.25169) 368 : cephadm 1 Detected new or changed devices on smithi078 2023-07-29T17:19:14.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cephadm 2023-07-29T17:19:12.653839+0000 mgr.y (mgr.25169) 369 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:19:14.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cluster 2023-07-29T17:19:12.655251+0000 mgr.y (mgr.25169) 370 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2023-07-29T17:19:14.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cluster 2023-07-29T17:19:12.655787+0000 mgr.y (mgr.25169) 371 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:14.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cephadm 2023-07-29T17:19:12.778636+0000 mgr.y (mgr.25169) 372 : cephadm 1 Upgrade: Setting container_image for all rgw 2023-07-29T17:19:14.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cephadm 2023-07-29T17:19:12.835247+0000 mgr.y (mgr.25169) 373 : cephadm 1 Upgrade: Setting container_image for all rbd-mirror 2023-07-29T17:19:14.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:14 smithi129 bash[53756]: cephadm 2023-07-29T17:19:12.849845+0000 mgr.y (mgr.25169) 374 : cephadm 1 Upgrade: Setting container_image for all ceph-exporter 2023-07-29T17:19:14.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cephadm 2023-07-29T17:19:12.623237+0000 mgr.y (mgr.25169) 368 : cephadm 1 Detected new or changed devices on smithi078 2023-07-29T17:19:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cephadm 2023-07-29T17:19:12.653839+0000 mgr.y (mgr.25169) 369 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:19:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cluster 2023-07-29T17:19:12.655251+0000 mgr.y (mgr.25169) 370 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2023-07-29T17:19:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cluster 2023-07-29T17:19:12.655787+0000 mgr.y (mgr.25169) 371 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cephadm 2023-07-29T17:19:12.778636+0000 mgr.y (mgr.25169) 372 : cephadm 1 Upgrade: Setting container_image for all rgw 2023-07-29T17:19:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cephadm 2023-07-29T17:19:12.835247+0000 mgr.y (mgr.25169) 373 : cephadm 1 Upgrade: Setting container_image for all rbd-mirror 2023-07-29T17:19:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[58614]: cephadm 2023-07-29T17:19:12.849845+0000 mgr.y (mgr.25169) 374 : cephadm 1 Upgrade: Setting container_image for all ceph-exporter 2023-07-29T17:19:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cephadm 2023-07-29T17:19:12.623237+0000 mgr.y (mgr.25169) 368 : cephadm 1 Detected new or changed devices on smithi078 2023-07-29T17:19:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cephadm 2023-07-29T17:19:12.653839+0000 mgr.y (mgr.25169) 369 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:19:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cluster 2023-07-29T17:19:12.655251+0000 mgr.y (mgr.25169) 370 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2023-07-29T17:19:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cluster 2023-07-29T17:19:12.655787+0000 mgr.y (mgr.25169) 371 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cephadm 2023-07-29T17:19:12.778636+0000 mgr.y (mgr.25169) 372 : cephadm 1 Upgrade: Setting container_image for all rgw 2023-07-29T17:19:14.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cephadm 2023-07-29T17:19:12.835247+0000 mgr.y (mgr.25169) 373 : cephadm 1 Upgrade: Setting container_image for all rbd-mirror 2023-07-29T17:19:14.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:14 smithi078 bash[60360]: cephadm 2023-07-29T17:19:12.849845+0000 mgr.y (mgr.25169) 374 : cephadm 1 Upgrade: Setting container_image for all ceph-exporter 2023-07-29T17:19:16.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:16 smithi129 bash[53756]: cluster 2023-07-29T17:19:14.657096+0000 mgr.y (mgr.25169) 375 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:16.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:16 smithi129 bash[53756]: cephadm 2023-07-29T17:19:15.128817+0000 mgr.y (mgr.25169) 376 : cephadm 1 Upgrade: Updating iscsi.foo.smithi129.vbvpri 2023-07-29T17:19:16.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:16 smithi129 bash[53756]: audit 2023-07-29T17:19:15.138359+0000 mon.a (mon.0) 803 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:16.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:16 smithi129 bash[53756]: audit 2023-07-29T17:19:15.140750+0000 mon.a (mon.0) 804 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:19:16.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:16 smithi129 bash[53756]: audit 2023-07-29T17:19:15.157322+0000 mon.a (mon.0) 805 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:16.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:16 smithi129 bash[53756]: cephadm 2023-07-29T17:19:15.159218+0000 mgr.y (mgr.25169) 377 : cephadm 1 Deploying daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:19:16.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[58614]: cluster 2023-07-29T17:19:14.657096+0000 mgr.y (mgr.25169) 375 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:16.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[58614]: cephadm 2023-07-29T17:19:15.128817+0000 mgr.y (mgr.25169) 376 : cephadm 1 Upgrade: Updating iscsi.foo.smithi129.vbvpri 2023-07-29T17:19:16.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[58614]: audit 2023-07-29T17:19:15.138359+0000 mon.a (mon.0) 803 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:16.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[58614]: audit 2023-07-29T17:19:15.140750+0000 mon.a (mon.0) 804 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:19:16.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[58614]: audit 2023-07-29T17:19:15.157322+0000 mon.a (mon.0) 805 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:16.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[58614]: cephadm 2023-07-29T17:19:15.159218+0000 mgr.y (mgr.25169) 377 : cephadm 1 Deploying daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:19:16.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[60360]: cluster 2023-07-29T17:19:14.657096+0000 mgr.y (mgr.25169) 375 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:16.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[60360]: cephadm 2023-07-29T17:19:15.128817+0000 mgr.y (mgr.25169) 376 : cephadm 1 Upgrade: Updating iscsi.foo.smithi129.vbvpri 2023-07-29T17:19:16.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[60360]: audit 2023-07-29T17:19:15.138359+0000 mon.a (mon.0) 803 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:16.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[60360]: audit 2023-07-29T17:19:15.140750+0000 mon.a (mon.0) 804 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi129.vbvpri", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-29T17:19:16.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[60360]: audit 2023-07-29T17:19:15.157322+0000 mon.a (mon.0) 805 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:16.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:16 smithi078 bash[60360]: cephadm 2023-07-29T17:19:15.159218+0000 mgr.y (mgr.25169) 377 : cephadm 1 Deploying daemon iscsi.foo.smithi129.vbvpri on smithi129 2023-07-29T17:19:18.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:18 smithi129 bash[53756]: cluster 2023-07-29T17:19:16.658088+0000 mgr.y (mgr.25169) 378 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:18.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:18 smithi129 bash[53756]: audit 2023-07-29T17:19:17.862899+0000 mon.a (mon.0) 806 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:18.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:18 smithi129 bash[53756]: audit 2023-07-29T17:19:17.875715+0000 mon.a (mon.0) 807 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:18.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:18 smithi078 bash[58614]: cluster 2023-07-29T17:19:16.658088+0000 mgr.y (mgr.25169) 378 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:18.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:18 smithi078 bash[58614]: audit 2023-07-29T17:19:17.862899+0000 mon.a (mon.0) 806 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:18.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:18 smithi078 bash[58614]: audit 2023-07-29T17:19:17.875715+0000 mon.a (mon.0) 807 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:18.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:18 smithi078 bash[60360]: cluster 2023-07-29T17:19:16.658088+0000 mgr.y (mgr.25169) 378 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:18.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:18 smithi078 bash[60360]: audit 2023-07-29T17:19:17.862899+0000 mon.a (mon.0) 806 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:18.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:18 smithi078 bash[60360]: audit 2023-07-29T17:19:17.875715+0000 mon.a (mon.0) 807 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:20.425 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:20 smithi129 bash[53756]: cluster 2023-07-29T17:19:18.659747+0000 mgr.y (mgr.25169) 379 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:20.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:20 smithi078 bash[58614]: cluster 2023-07-29T17:19:18.659747+0000 mgr.y (mgr.25169) 379 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:20.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:20 smithi078 bash[60360]: cluster 2023-07-29T17:19:18.659747+0000 mgr.y (mgr.25169) 379 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:21.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:21 smithi078 bash[58614]: audit 2023-07-29T17:19:20.204689+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.129:0/2265733232' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-29T17:19:21.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:21 smithi078 bash[58614]: audit 2023-07-29T17:19:20.583141+0000 mon.a (mon.0) 808 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: dispatch 2023-07-29T17:19:21.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:21 smithi078 bash[58614]: audit 2023-07-29T17:19:20.584375+0000 mon.b (mon.1) 12 : audit 1 from='client.? 172.21.15.129:0/3072204668' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: dispatch 2023-07-29T17:19:21.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:21 smithi078 bash[60360]: audit 2023-07-29T17:19:20.204689+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.129:0/2265733232' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-29T17:19:21.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:21 smithi078 bash[60360]: audit 2023-07-29T17:19:20.583141+0000 mon.a (mon.0) 808 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: dispatch 2023-07-29T17:19:21.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:21 smithi078 bash[60360]: audit 2023-07-29T17:19:20.584375+0000 mon.b (mon.1) 12 : audit 1 from='client.? 172.21.15.129:0/3072204668' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: dispatch 2023-07-29T17:19:21.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:21 smithi129 bash[53756]: audit 2023-07-29T17:19:20.204689+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.129:0/2265733232' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-29T17:19:21.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:21 smithi129 bash[53756]: audit 2023-07-29T17:19:20.583141+0000 mon.a (mon.0) 808 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: dispatch 2023-07-29T17:19:21.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:21 smithi129 bash[53756]: audit 2023-07-29T17:19:20.584375+0000 mon.b (mon.1) 12 : audit 1 from='client.? 172.21.15.129:0/3072204668' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]: dispatch 2023-07-29T17:19:22.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[60360]: cluster 2023-07-29T17:19:20.660915+0000 mgr.y (mgr.25169) 380 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:22.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[60360]: audit 2023-07-29T17:19:21.168192+0000 mon.a (mon.0) 809 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]': finished 2023-07-29T17:19:22.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[60360]: cluster 2023-07-29T17:19:21.168337+0000 mon.a (mon.0) 810 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2023-07-29T17:19:22.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[60360]: audit 2023-07-29T17:19:21.556450+0000 mon.a (mon.0) 811 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]: dispatch 2023-07-29T17:19:22.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[60360]: audit 2023-07-29T17:19:21.557956+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.129:0/128091418' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]: dispatch 2023-07-29T17:19:22.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[58614]: cluster 2023-07-29T17:19:20.660915+0000 mgr.y (mgr.25169) 380 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:22.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[58614]: audit 2023-07-29T17:19:21.168192+0000 mon.a (mon.0) 809 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]': finished 2023-07-29T17:19:22.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[58614]: cluster 2023-07-29T17:19:21.168337+0000 mon.a (mon.0) 810 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2023-07-29T17:19:22.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[58614]: audit 2023-07-29T17:19:21.556450+0000 mon.a (mon.0) 811 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]: dispatch 2023-07-29T17:19:22.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:22 smithi078 bash[58614]: audit 2023-07-29T17:19:21.557956+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.129:0/128091418' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]: dispatch 2023-07-29T17:19:22.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:22 smithi129 bash[53756]: cluster 2023-07-29T17:19:20.660915+0000 mgr.y (mgr.25169) 380 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:22.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:22 smithi129 bash[53756]: audit 2023-07-29T17:19:21.168192+0000 mon.a (mon.0) 809 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/1179635431"}]': finished 2023-07-29T17:19:22.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:22 smithi129 bash[53756]: cluster 2023-07-29T17:19:21.168337+0000 mon.a (mon.0) 810 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2023-07-29T17:19:22.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:22 smithi129 bash[53756]: audit 2023-07-29T17:19:21.556450+0000 mon.a (mon.0) 811 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]: dispatch 2023-07-29T17:19:22.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:22 smithi129 bash[53756]: audit 2023-07-29T17:19:21.557956+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.129:0/128091418' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]: dispatch 2023-07-29T17:19:23.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[58614]: audit 2023-07-29T17:19:22.173199+0000 mon.a (mon.0) 812 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]': finished 2023-07-29T17:19:23.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[58614]: cluster 2023-07-29T17:19:22.173329+0000 mon.a (mon.0) 813 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2023-07-29T17:19:23.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[58614]: audit 2023-07-29T17:19:22.572324+0000 mon.a (mon.0) 814 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]: dispatch 2023-07-29T17:19:23.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[58614]: audit 2023-07-29T17:19:22.573407+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.129:0/415988268' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]: dispatch 2023-07-29T17:19:23.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[60360]: audit 2023-07-29T17:19:22.173199+0000 mon.a (mon.0) 812 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]': finished 2023-07-29T17:19:23.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[60360]: cluster 2023-07-29T17:19:22.173329+0000 mon.a (mon.0) 813 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2023-07-29T17:19:23.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[60360]: audit 2023-07-29T17:19:22.572324+0000 mon.a (mon.0) 814 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]: dispatch 2023-07-29T17:19:23.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:23 smithi078 bash[60360]: audit 2023-07-29T17:19:22.573407+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.129:0/415988268' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]: dispatch 2023-07-29T17:19:23.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:23 smithi129 bash[53756]: audit 2023-07-29T17:19:22.173199+0000 mon.a (mon.0) 812 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2276122683"}]': finished 2023-07-29T17:19:23.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:23 smithi129 bash[53756]: cluster 2023-07-29T17:19:22.173329+0000 mon.a (mon.0) 813 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2023-07-29T17:19:23.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:23 smithi129 bash[53756]: audit 2023-07-29T17:19:22.572324+0000 mon.a (mon.0) 814 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]: dispatch 2023-07-29T17:19:23.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:23 smithi129 bash[53756]: audit 2023-07-29T17:19:22.573407+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.129:0/415988268' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]: dispatch 2023-07-29T17:19:24.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: cluster 2023-07-29T17:19:22.661900+0000 mgr.y (mgr.25169) 381 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:24.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: audit 2023-07-29T17:19:23.182545+0000 mon.a (mon.0) 815 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]': finished 2023-07-29T17:19:24.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: cluster 2023-07-29T17:19:23.182635+0000 mon.a (mon.0) 816 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-07-29T17:19:24.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: audit 2023-07-29T17:19:23.585325+0000 mon.a (mon.0) 817 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]: dispatch 2023-07-29T17:19:24.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: audit 2023-07-29T17:19:23.586598+0000 mon.b (mon.1) 15 : audit 1 from='client.? 172.21.15.129:0/4028790375' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]: dispatch 2023-07-29T17:19:24.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: audit 2023-07-29T17:19:23.976078+0000 mon.a (mon.0) 818 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:24.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:24 smithi129 bash[53756]: audit 2023-07-29T17:19:23.986322+0000 mon.a (mon.0) 819 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:24.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: cluster 2023-07-29T17:19:22.661900+0000 mgr.y (mgr.25169) 381 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:24.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: audit 2023-07-29T17:19:23.182545+0000 mon.a (mon.0) 815 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]': finished 2023-07-29T17:19:24.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: cluster 2023-07-29T17:19:23.182635+0000 mon.a (mon.0) 816 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-07-29T17:19:24.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: audit 2023-07-29T17:19:23.585325+0000 mon.a (mon.0) 817 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]: dispatch 2023-07-29T17:19:24.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: audit 2023-07-29T17:19:23.586598+0000 mon.b (mon.1) 15 : audit 1 from='client.? 172.21.15.129:0/4028790375' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]: dispatch 2023-07-29T17:19:24.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: audit 2023-07-29T17:19:23.976078+0000 mon.a (mon.0) 818 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:24.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[58614]: audit 2023-07-29T17:19:23.986322+0000 mon.a (mon.0) 819 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:24.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: cluster 2023-07-29T17:19:22.661900+0000 mgr.y (mgr.25169) 381 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail 2023-07-29T17:19:24.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: audit 2023-07-29T17:19:23.182545+0000 mon.a (mon.0) 815 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6833/79426902"}]': finished 2023-07-29T17:19:24.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: cluster 2023-07-29T17:19:23.182635+0000 mon.a (mon.0) 816 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-07-29T17:19:24.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: audit 2023-07-29T17:19:23.585325+0000 mon.a (mon.0) 817 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]: dispatch 2023-07-29T17:19:24.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: audit 2023-07-29T17:19:23.586598+0000 mon.b (mon.1) 15 : audit 1 from='client.? 172.21.15.129:0/4028790375' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]: dispatch 2023-07-29T17:19:24.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: audit 2023-07-29T17:19:23.976078+0000 mon.a (mon.0) 818 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:24.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:24 smithi078 bash[60360]: audit 2023-07-29T17:19:23.986322+0000 mon.a (mon.0) 819 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:25.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:25 smithi129 bash[53756]: audit 2023-07-29T17:19:24.194003+0000 mon.a (mon.0) 820 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]': finished 2023-07-29T17:19:25.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:25 smithi129 bash[53756]: cluster 2023-07-29T17:19:24.194127+0000 mon.a (mon.0) 821 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-07-29T17:19:25.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:25 smithi129 bash[53756]: audit 2023-07-29T17:19:24.592884+0000 mon.a (mon.0) 822 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]: dispatch 2023-07-29T17:19:25.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:25 smithi129 bash[53756]: audit 2023-07-29T17:19:24.594154+0000 mon.b (mon.1) 16 : audit 1 from='client.? 172.21.15.129:0/2574141489' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]: dispatch 2023-07-29T17:19:25.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[60360]: audit 2023-07-29T17:19:24.194003+0000 mon.a (mon.0) 820 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]': finished 2023-07-29T17:19:25.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[60360]: cluster 2023-07-29T17:19:24.194127+0000 mon.a (mon.0) 821 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-07-29T17:19:25.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[60360]: audit 2023-07-29T17:19:24.592884+0000 mon.a (mon.0) 822 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]: dispatch 2023-07-29T17:19:25.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[60360]: audit 2023-07-29T17:19:24.594154+0000 mon.b (mon.1) 16 : audit 1 from='client.? 172.21.15.129:0/2574141489' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]: dispatch 2023-07-29T17:19:25.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[58614]: audit 2023-07-29T17:19:24.194003+0000 mon.a (mon.0) 820 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/2261738764"}]': finished 2023-07-29T17:19:25.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[58614]: cluster 2023-07-29T17:19:24.194127+0000 mon.a (mon.0) 821 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-07-29T17:19:25.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[58614]: audit 2023-07-29T17:19:24.592884+0000 mon.a (mon.0) 822 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]: dispatch 2023-07-29T17:19:25.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:25 smithi078 bash[58614]: audit 2023-07-29T17:19:24.594154+0000 mon.b (mon.1) 16 : audit 1 from='client.? 172.21.15.129:0/2574141489' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]: dispatch 2023-07-29T17:19:26.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: cluster 2023-07-29T17:19:24.663057+0000 mgr.y (mgr.25169) 382 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:19:26.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:25.206551+0000 mon.a (mon.0) 823 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]': finished 2023-07-29T17:19:26.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: cluster 2023-07-29T17:19:25.206634+0000 mon.a (mon.0) 824 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2023-07-29T17:19:26.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:25.282815+0000 mon.a (mon.0) 825 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:25.293595+0000 mon.a (mon.0) 826 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:25.588875+0000 mon.a (mon.0) 827 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: dispatch 2023-07-29T17:19:26.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:25.590030+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.129:0/2811866007' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: dispatch 2023-07-29T17:19:26.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:26.101871+0000 mon.a (mon.0) 828 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:26 smithi129 bash[53756]: audit 2023-07-29T17:19:26.111796+0000 mon.a (mon.0) 829 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: cluster 2023-07-29T17:19:24.663057+0000 mgr.y (mgr.25169) 382 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:25.206551+0000 mon.a (mon.0) 823 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]': finished 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: cluster 2023-07-29T17:19:25.206634+0000 mon.a (mon.0) 824 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:25.282815+0000 mon.a (mon.0) 825 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:25.293595+0000 mon.a (mon.0) 826 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:25.588875+0000 mon.a (mon.0) 827 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: dispatch 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:25.590030+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.129:0/2811866007' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: dispatch 2023-07-29T17:19:26.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:26.101871+0000 mon.a (mon.0) 828 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[58614]: audit 2023-07-29T17:19:26.111796+0000 mon.a (mon.0) 829 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: cluster 2023-07-29T17:19:24.663057+0000 mgr.y (mgr.25169) 382 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:19:26.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:25.206551+0000 mon.a (mon.0) 823 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:0/103962009"}]': finished 2023-07-29T17:19:26.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: cluster 2023-07-29T17:19:25.206634+0000 mon.a (mon.0) 824 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2023-07-29T17:19:26.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:25.282815+0000 mon.a (mon.0) 825 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:25.293595+0000 mon.a (mon.0) 826 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:25.588875+0000 mon.a (mon.0) 827 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: dispatch 2023-07-29T17:19:26.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:25.590030+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.129:0/2811866007' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]: dispatch 2023-07-29T17:19:26.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:26.101871+0000 mon.a (mon.0) 828 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:26.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:26 smithi078 bash[60360]: audit 2023-07-29T17:19:26.111796+0000 mon.a (mon.0) 829 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:27.675 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:27 smithi129 bash[53756]: audit 2023-07-29T17:19:26.299825+0000 mon.a (mon.0) 830 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]': finished 2023-07-29T17:19:27.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:27 smithi129 bash[53756]: cluster 2023-07-29T17:19:26.300113+0000 mon.a (mon.0) 831 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2023-07-29T17:19:27.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:27 smithi129 bash[53756]: cluster 2023-07-29T17:19:26.663873+0000 mgr.y (mgr.25169) 383 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:19:27.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:27 smithi078 bash[58614]: audit 2023-07-29T17:19:26.299825+0000 mon.a (mon.0) 830 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]': finished 2023-07-29T17:19:27.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:27 smithi078 bash[58614]: cluster 2023-07-29T17:19:26.300113+0000 mon.a (mon.0) 831 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2023-07-29T17:19:27.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:27 smithi078 bash[58614]: cluster 2023-07-29T17:19:26.663873+0000 mgr.y (mgr.25169) 383 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:19:27.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:27 smithi078 bash[60360]: audit 2023-07-29T17:19:26.299825+0000 mon.a (mon.0) 830 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi129.vbvpri' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.129:6832/79426902"}]': finished 2023-07-29T17:19:27.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:27 smithi078 bash[60360]: cluster 2023-07-29T17:19:26.300113+0000 mon.a (mon.0) 831 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2023-07-29T17:19:27.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:27 smithi078 bash[60360]: cluster 2023-07-29T17:19:26.663873+0000 mgr.y (mgr.25169) 383 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-07-29T17:19:30.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:29 smithi129 bash[53756]: cluster 2023-07-29T17:19:28.665409+0000 mgr.y (mgr.25169) 384 : cluster 0 pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2023-07-29T17:19:30.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:29 smithi078 bash[58614]: cluster 2023-07-29T17:19:28.665409+0000 mgr.y (mgr.25169) 384 : cluster 0 pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2023-07-29T17:19:30.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:29 smithi078 bash[60360]: cluster 2023-07-29T17:19:28.665409+0000 mgr.y (mgr.25169) 384 : cluster 0 pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2023-07-29T17:19:31.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:30 smithi129 bash[53756]: audit 2023-07-29T17:19:29.857484+0000 mgr.y (mgr.25169) 385 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:31.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:30 smithi078 bash[58614]: audit 2023-07-29T17:19:29.857484+0000 mgr.y (mgr.25169) 385 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:31.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:30 smithi078 bash[60360]: audit 2023-07-29T17:19:29.857484+0000 mgr.y (mgr.25169) 385 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:32.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:31 smithi129 bash[53756]: cluster 2023-07-29T17:19:30.666645+0000 mgr.y (mgr.25169) 386 : cluster 0 pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2023-07-29T17:19:32.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:31 smithi078 bash[58614]: cluster 2023-07-29T17:19:30.666645+0000 mgr.y (mgr.25169) 386 : cluster 0 pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2023-07-29T17:19:32.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:31 smithi078 bash[60360]: cluster 2023-07-29T17:19:30.666645+0000 mgr.y (mgr.25169) 386 : cluster 0 pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2023-07-29T17:19:34.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:33 smithi129 bash[53756]: cluster 2023-07-29T17:19:32.667952+0000 mgr.y (mgr.25169) 387 : cluster 0 pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-29T17:19:34.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:33 smithi078 bash[58614]: cluster 2023-07-29T17:19:32.667952+0000 mgr.y (mgr.25169) 387 : cluster 0 pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-29T17:19:34.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:33 smithi078 bash[60360]: cluster 2023-07-29T17:19:32.667952+0000 mgr.y (mgr.25169) 387 : cluster 0 pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-29T17:19:36.175 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:35 smithi129 bash[53756]: cluster 2023-07-29T17:19:34.669330+0000 mgr.y (mgr.25169) 388 : cluster 0 pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-29T17:19:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:35 smithi078 bash[58614]: cluster 2023-07-29T17:19:34.669330+0000 mgr.y (mgr.25169) 388 : cluster 0 pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-29T17:19:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:35 smithi078 bash[60360]: cluster 2023-07-29T17:19:34.669330+0000 mgr.y (mgr.25169) 388 : cluster 0 pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-29T17:19:36.906 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:19:37.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:35.868701+0000 mon.a (mon.0) 832 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:35.878501+0000 mon.a (mon.0) 833 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:35.880449+0000 mon.a (mon.0) 834 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:35.882004+0000 mon.a (mon.0) 835 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: cephadm 2023-07-29T17:19:35.883351+0000 mgr.y (mgr.25169) 389 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: cluster 2023-07-29T17:19:35.884555+0000 mgr.y (mgr.25169) 390 : cluster 0 pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: cluster 2023-07-29T17:19:35.884980+0000 mgr.y (mgr.25169) 391 : cluster 0 pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: cephadm 2023-07-29T17:19:35.889671+0000 mgr.y (mgr.25169) 392 : cephadm 1 Checking dashboard <-> RGW credentials 2023-07-29T17:19:37.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.372978+0000 mon.a (mon.0) 836 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.390221+0000 mon.a (mon.0) 837 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.402742+0000 mon.a (mon.0) 838 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.521727+0000 mon.a (mon.0) 839 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.542519+0000 mon.a (mon.0) 840 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.545281+0000 mon.a (mon.0) 841 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.547882+0000 mon.a (mon.0) 842 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:19:37.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.556143+0000 mon.a (mon.0) 843 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.568865+0000 mon.a (mon.0) 844 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.573233+0000 mon.a (mon.0) 845 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.575555+0000 mon.a (mon.0) 846 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.577077+0000 mon.a (mon.0) 847 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.579704+0000 mon.a (mon.0) 848 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.582751+0000 mon.a (mon.0) 849 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.584973+0000 mon.a (mon.0) 850 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.586585+0000 mon.a (mon.0) 851 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.588245+0000 mon.a (mon.0) 852 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.589837+0000 mon.a (mon.0) 853 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.591615+0000 mon.a (mon.0) 854 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.603182+0000 mon.a (mon.0) 855 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.604473+0000 mon.a (mon.0) 856 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi129.vbvpri"}]: dispatch 2023-07-29T17:19:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.611115+0000 mon.a (mon.0) 857 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi129.vbvpri"}]': finished 2023-07-29T17:19:37.181 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.613062+0000 mon.a (mon.0) 858 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.181 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:36 smithi129 bash[53756]: audit 2023-07-29T17:19:36.622859+0000 mon.a (mon.0) 859 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:35.868701+0000 mon.a (mon.0) 832 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:35.878501+0000 mon.a (mon.0) 833 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:35.880449+0000 mon.a (mon.0) 834 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:35.882004+0000 mon.a (mon.0) 835 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: cephadm 2023-07-29T17:19:35.883351+0000 mgr.y (mgr.25169) 389 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: cluster 2023-07-29T17:19:35.884555+0000 mgr.y (mgr.25169) 390 : cluster 0 pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: cluster 2023-07-29T17:19:35.884980+0000 mgr.y (mgr.25169) 391 : cluster 0 pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-29T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: cephadm 2023-07-29T17:19:35.889671+0000 mgr.y (mgr.25169) 392 : cephadm 1 Checking dashboard <-> RGW credentials 2023-07-29T17:19:37.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.372978+0000 mon.a (mon.0) 836 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.390221+0000 mon.a (mon.0) 837 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.402742+0000 mon.a (mon.0) 838 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.521727+0000 mon.a (mon.0) 839 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.542519+0000 mon.a (mon.0) 840 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.545281+0000 mon.a (mon.0) 841 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.547882+0000 mon.a (mon.0) 842 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.556143+0000 mon.a (mon.0) 843 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.568865+0000 mon.a (mon.0) 844 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.573233+0000 mon.a (mon.0) 845 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.575555+0000 mon.a (mon.0) 846 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.577077+0000 mon.a (mon.0) 847 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:35.868701+0000 mon.a (mon.0) 832 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:35.878501+0000 mon.a (mon.0) 833 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:35.880449+0000 mon.a (mon.0) 834 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:19:37.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:35.882004+0000 mon.a (mon.0) 835 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:19:37.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: cephadm 2023-07-29T17:19:35.883351+0000 mgr.y (mgr.25169) 389 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:19:37.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: cluster 2023-07-29T17:19:35.884555+0000 mgr.y (mgr.25169) 390 : cluster 0 pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:19:37.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: cluster 2023-07-29T17:19:35.884980+0000 mgr.y (mgr.25169) 391 : cluster 0 pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-29T17:19:37.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: cephadm 2023-07-29T17:19:35.889671+0000 mgr.y (mgr.25169) 392 : cephadm 1 Checking dashboard <-> RGW credentials 2023-07-29T17:19:37.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.372978+0000 mon.a (mon.0) 836 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.390221+0000 mon.a (mon.0) 837 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.402742+0000 mon.a (mon.0) 838 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.521727+0000 mon.a (mon.0) 839 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.542519+0000 mon.a (mon.0) 840 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.545281+0000 mon.a (mon.0) 841 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.547882+0000 mon.a (mon.0) 842 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.556143+0000 mon.a (mon.0) 843 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.568865+0000 mon.a (mon.0) 844 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:19:37.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.573233+0000 mon.a (mon.0) 845 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.575555+0000 mon.a (mon.0) 846 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.577077+0000 mon.a (mon.0) 847 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.579704+0000 mon.a (mon.0) 848 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.582751+0000 mon.a (mon.0) 849 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.584973+0000 mon.a (mon.0) 850 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.586585+0000 mon.a (mon.0) 851 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.588245+0000 mon.a (mon.0) 852 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.589837+0000 mon.a (mon.0) 853 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.591615+0000 mon.a (mon.0) 854 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.603182+0000 mon.a (mon.0) 855 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.604473+0000 mon.a (mon.0) 856 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi129.vbvpri"}]: dispatch 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.611115+0000 mon.a (mon.0) 857 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi129.vbvpri"}]': finished 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.613062+0000 mon.a (mon.0) 858 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[60360]: audit 2023-07-29T17:19:36.622859+0000 mon.a (mon.0) 859 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.579704+0000 mon.a (mon.0) 848 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.582751+0000 mon.a (mon.0) 849 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.584973+0000 mon.a (mon.0) 850 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.586585+0000 mon.a (mon.0) 851 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.588245+0000 mon.a (mon.0) 852 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.589837+0000 mon.a (mon.0) 853 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.591615+0000 mon.a (mon.0) 854 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.603182+0000 mon.a (mon.0) 855 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.604473+0000 mon.a (mon.0) 856 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi129.vbvpri"}]: dispatch 2023-07-29T17:19:37.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.611115+0000 mon.a (mon.0) 857 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi129.vbvpri"}]': finished 2023-07-29T17:19:37.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.613062+0000 mon.a (mon.0) 858 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:37.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:36 smithi078 bash[58614]: audit 2023-07-29T17:19:36.622859+0000 mon.a (mon.0) 859 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:37.644 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:19:37.645 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (9m) 13s ago 15m 10.8M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:19:37.645 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (8m) 12s ago 15m 26.6M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:19:37.645 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (19s) 12s ago 13m 49.0M - 3.6 b662b4d80c86 a64a43bc0b16 2023-07-29T17:19:37.645 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (7m) 12s ago 20m 430M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:19:37.645 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (8m) 13s ago 21m 479M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:19:37.645 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7m) 13s ago 21m 80.5M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (6m) 12s ago 20m 59.7M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6m) 13s ago 20m 59.8M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (9m) 13s ago 15m 5971k - 0.18.1 e5a616e4b9cf 14d1d94809f3 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (9m) 12s ago 15m 3976k - 0.18.1 e5a616e4b9cf 9a31bf155d15 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4m) 13s ago 19m 61.6M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4m) 13s ago 19m 64.1M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3m) 13s ago 18m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:19:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3m) 13s ago 18m 60.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (2m) 12s ago 18m 60.7M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (2m) 12s ago 17m 54.1M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (116s) 12s ago 17m 53.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (86s) 12s ago 16m 55.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 12s ago 16m - - 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (62s) 13s ago 13m 83.7M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (48s) 12s ago 13m 83.7M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:19:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (55s) 13s ago 13m 83.8M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:19:37.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (42s) 12s ago 13m 83.7M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:19:38.083 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:19:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:19:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:19:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:19:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:19:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:19:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:19:38.086 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:19:38.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:19:38.086 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:19:38.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: audit 2023-07-29T17:19:36.523101+0000 mgr.y (mgr.25169) 393 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:19:38.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: cephadm 2023-07-29T17:19:36.544756+0000 mgr.y (mgr.25169) 394 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.129:5000 to Dashboard 2023-07-29T17:19:38.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: audit 2023-07-29T17:19:36.545836+0000 mgr.y (mgr.25169) 395 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:19:38.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: audit 2023-07-29T17:19:36.548613+0000 mgr.y (mgr.25169) 396 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:19:38.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: cephadm 2023-07-29T17:19:36.592574+0000 mgr.y (mgr.25169) 397 : cephadm 1 Upgrade: Setting container_image for all iscsi 2023-07-29T17:19:38.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: cephadm 2023-07-29T17:19:36.614007+0000 mgr.y (mgr.25169) 398 : cephadm 1 Upgrade: Setting container_image for all nfs 2023-07-29T17:19:38.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: audit 2023-07-29T17:19:36.864663+0000 mgr.y (mgr.25169) 399 : audit 0 from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:38.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:37 smithi129 bash[53756]: cluster 2023-07-29T17:19:36.877812+0000 mon.a (mon.0) 860 : cluster 3 Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:19:38.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: audit 2023-07-29T17:19:36.523101+0000 mgr.y (mgr.25169) 393 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:19:38.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: cephadm 2023-07-29T17:19:36.544756+0000 mgr.y (mgr.25169) 394 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.129:5000 to Dashboard 2023-07-29T17:19:38.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: audit 2023-07-29T17:19:36.545836+0000 mgr.y (mgr.25169) 395 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: audit 2023-07-29T17:19:36.548613+0000 mgr.y (mgr.25169) 396 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: cephadm 2023-07-29T17:19:36.592574+0000 mgr.y (mgr.25169) 397 : cephadm 1 Upgrade: Setting container_image for all iscsi 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: cephadm 2023-07-29T17:19:36.614007+0000 mgr.y (mgr.25169) 398 : cephadm 1 Upgrade: Setting container_image for all nfs 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: audit 2023-07-29T17:19:36.864663+0000 mgr.y (mgr.25169) 399 : audit 0 from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[58614]: cluster 2023-07-29T17:19:36.877812+0000 mon.a (mon.0) 860 : cluster 3 Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: audit 2023-07-29T17:19:36.523101+0000 mgr.y (mgr.25169) 393 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-29T17:19:38.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: cephadm 2023-07-29T17:19:36.544756+0000 mgr.y (mgr.25169) 394 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.129:5000 to Dashboard 2023-07-29T17:19:38.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: audit 2023-07-29T17:19:36.545836+0000 mgr.y (mgr.25169) 395 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-29T17:19:38.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: audit 2023-07-29T17:19:36.548613+0000 mgr.y (mgr.25169) 396 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi129"}]: dispatch 2023-07-29T17:19:38.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: cephadm 2023-07-29T17:19:36.592574+0000 mgr.y (mgr.25169) 397 : cephadm 1 Upgrade: Setting container_image for all iscsi 2023-07-29T17:19:38.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: cephadm 2023-07-29T17:19:36.614007+0000 mgr.y (mgr.25169) 398 : cephadm 1 Upgrade: Setting container_image for all nfs 2023-07-29T17:19:38.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: audit 2023-07-29T17:19:36.864663+0000 mgr.y (mgr.25169) 399 : audit 0 from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:38.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:37 smithi078 bash[60360]: cluster 2023-07-29T17:19:36.877812+0000 mon.a (mon.0) 860 : cluster 3 Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-29T17:19:38.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:19:38.461 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:19:38.461 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:19:38.461 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:19:38.461 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: "osd", 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: "iscsi", 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:19:38.462 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "18/23 daemons upgraded", 2023-07-29T17:19:38.463 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading node-exporter daemons", 2023-07-29T17:19:38.463 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:19:38.463 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:19:38.895 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-29T17:19:38.895 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-29T17:19:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:19:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:38 smithi129 bash[53756]: audit 2023-07-29T17:19:37.240352+0000 mgr.y (mgr.25169) 400 : audit 0 from='client.44747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:38 smithi129 bash[53756]: audit 2023-07-29T17:19:37.595794+0000 mgr.y (mgr.25169) 401 : audit 0 from='client.44753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:38 smithi129 bash[53756]: cluster 2023-07-29T17:19:37.886275+0000 mgr.y (mgr.25169) 402 : cluster 0 pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-29T17:19:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:38 smithi129 bash[53756]: audit 2023-07-29T17:19:38.059781+0000 mon.c (mon.2) 15 : audit 0 from='client.44759 172.21.15.78:0/521901036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:39.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[58614]: audit 2023-07-29T17:19:37.240352+0000 mgr.y (mgr.25169) 400 : audit 0 from='client.44747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:39.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[58614]: audit 2023-07-29T17:19:37.595794+0000 mgr.y (mgr.25169) 401 : audit 0 from='client.44753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:39.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[58614]: cluster 2023-07-29T17:19:37.886275+0000 mgr.y (mgr.25169) 402 : cluster 0 pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-29T17:19:39.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[58614]: audit 2023-07-29T17:19:38.059781+0000 mon.c (mon.2) 15 : audit 0 from='client.44759 172.21.15.78:0/521901036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[60360]: audit 2023-07-29T17:19:37.240352+0000 mgr.y (mgr.25169) 400 : audit 0 from='client.44747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:39.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[60360]: audit 2023-07-29T17:19:37.595794+0000 mgr.y (mgr.25169) 401 : audit 0 from='client.44753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[60360]: cluster 2023-07-29T17:19:37.886275+0000 mgr.y (mgr.25169) 402 : cluster 0 pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-29T17:19:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:38 smithi078 bash[60360]: audit 2023-07-29T17:19:38.059781+0000 mon.c (mon.2) 15 : audit 0 from='client.44759 172.21.15.78:0/521901036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:19:39.695 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:39 smithi078 systemd[1]: Stopping Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:19:40.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:39 smithi129 bash[53756]: audit 2023-07-29T17:19:38.436973+0000 mgr.y (mgr.25169) 403 : audit 0 from='client.44765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:40.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:39 smithi129 bash[53756]: audit 2023-07-29T17:19:38.872081+0000 mon.a (mon.0) 861 : audit 0 from='client.34929 172.21.15.78:0/3522678427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:19:40.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:39 smithi129 bash[53756]: cephadm 2023-07-29T17:19:39.043945+0000 mgr.y (mgr.25169) 404 : cephadm 1 Upgrade: Updating node-exporter.a (1/2) 2023-07-29T17:19:40.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:39 smithi129 bash[53756]: cephadm 2023-07-29T17:19:39.044703+0000 mgr.y (mgr.25169) 405 : cephadm 1 Deploying daemon node-exporter.a on smithi078 2023-07-29T17:19:40.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[60360]: audit 2023-07-29T17:19:38.436973+0000 mgr.y (mgr.25169) 403 : audit 0 from='client.44765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:40.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[60360]: audit 2023-07-29T17:19:38.872081+0000 mon.a (mon.0) 861 : audit 0 from='client.34929 172.21.15.78:0/3522678427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:19:40.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[60360]: cephadm 2023-07-29T17:19:39.043945+0000 mgr.y (mgr.25169) 404 : cephadm 1 Upgrade: Updating node-exporter.a (1/2) 2023-07-29T17:19:40.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[60360]: cephadm 2023-07-29T17:19:39.044703+0000 mgr.y (mgr.25169) 405 : cephadm 1 Deploying daemon node-exporter.a on smithi078 2023-07-29T17:19:40.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[58614]: audit 2023-07-29T17:19:38.436973+0000 mgr.y (mgr.25169) 403 : audit 0 from='client.44765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:19:40.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[58614]: audit 2023-07-29T17:19:38.872081+0000 mon.a (mon.0) 861 : audit 0 from='client.34929 172.21.15.78:0/3522678427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:19:40.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[58614]: cephadm 2023-07-29T17:19:39.043945+0000 mgr.y (mgr.25169) 404 : cephadm 1 Upgrade: Updating node-exporter.a (1/2) 2023-07-29T17:19:40.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:39 smithi078 bash[58614]: cephadm 2023-07-29T17:19:39.044703+0000 mgr.y (mgr.25169) 405 : cephadm 1 Deploying daemon node-exporter.a on smithi078 2023-07-29T17:19:40.853 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[76854]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-node-exporter.a 2023-07-29T17:19:40.853 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:40 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:19:40.854 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:40 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-29T17:19:40.854 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:40 smithi078 systemd[1]: Stopped Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:19:40.854 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:40 smithi078 systemd[1]: Started Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:19:41.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:40 smithi129 bash[53756]: audit 2023-07-29T17:19:39.860357+0000 mgr.y (mgr.25169) 406 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:41.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:40 smithi129 bash[53756]: cluster 2023-07-29T17:19:39.887610+0000 mgr.y (mgr.25169) 407 : cluster 0 pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:41.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:40 smithi129 bash[53756]: audit 2023-07-29T17:19:40.863265+0000 mon.a (mon.0) 862 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:41.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:40 smithi129 bash[53756]: audit 2023-07-29T17:19:40.869578+0000 mon.a (mon.0) 863 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:41.194 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[77020]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2023-07-29T17:19:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[58614]: audit 2023-07-29T17:19:39.860357+0000 mgr.y (mgr.25169) 406 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[58614]: cluster 2023-07-29T17:19:39.887610+0000 mgr.y (mgr.25169) 407 : cluster 0 pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[58614]: audit 2023-07-29T17:19:40.863265+0000 mon.a (mon.0) 862 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[58614]: audit 2023-07-29T17:19:40.869578+0000 mon.a (mon.0) 863 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[60360]: audit 2023-07-29T17:19:39.860357+0000 mgr.y (mgr.25169) 406 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[60360]: cluster 2023-07-29T17:19:39.887610+0000 mgr.y (mgr.25169) 407 : cluster 0 pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[60360]: audit 2023-07-29T17:19:40.863265+0000 mon.a (mon.0) 862 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:40 smithi078 bash[60360]: audit 2023-07-29T17:19:40.869578+0000 mon.a (mon.0) 863 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:41.860 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:41 smithi078 bash[77020]: v1.5.0: Pulling from prometheus/node-exporter 2023-07-29T17:19:42.164 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:41 smithi078 bash[77020]: 22b70bddd3ac: Pulling fs layer 2023-07-29T17:19:42.164 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:41 smithi078 bash[77020]: 5c12815fee55: Pulling fs layer 2023-07-29T17:19:42.164 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:41 smithi078 bash[77020]: c0e87333d380: Pulling fs layer 2023-07-29T17:19:42.444 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: 5c12815fee55: Verifying Checksum 2023-07-29T17:19:42.445 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: 5c12815fee55: Download complete 2023-07-29T17:19:42.445 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: 22b70bddd3ac: Verifying Checksum 2023-07-29T17:19:42.445 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: 22b70bddd3ac: Download complete 2023-07-29T17:19:42.445 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: 22b70bddd3ac: Pull complete 2023-07-29T17:19:42.445 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: c0e87333d380: Verifying Checksum 2023-07-29T17:19:42.445 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: c0e87333d380: Download complete 2023-07-29T17:19:43.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[58614]: cluster 2023-07-29T17:19:41.888676+0000 mgr.y (mgr.25169) 408 : cluster 0 pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:43.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[60360]: cluster 2023-07-29T17:19:41.888676+0000 mgr.y (mgr.25169) 408 : cluster 0 pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:43.103 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:42 smithi078 bash[77020]: 5c12815fee55: Pull complete 2023-07-29T17:19:43.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:42 smithi129 bash[53756]: cluster 2023-07-29T17:19:41.888676+0000 mgr.y (mgr.25169) 408 : cluster 0 pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:43.389 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:43 smithi078 bash[77020]: c0e87333d380: Pull complete 2023-07-29T17:19:43.389 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:43 smithi078 bash[77020]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2023-07-29T17:19:43.389 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:43 smithi078 bash[77020]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2023-07-29T17:19:43.926 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:43 smithi129 systemd[1]: Stopping Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:19:45.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:44 smithi129 bash[53756]: cephadm 2023-07-29T17:19:43.332618+0000 mgr.y (mgr.25169) 409 : cephadm 1 Upgrade: Updating node-exporter.b (2/2) 2023-07-29T17:19:45.163 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:44 smithi129 bash[53756]: cephadm 2023-07-29T17:19:43.333363+0000 mgr.y (mgr.25169) 410 : cephadm 1 Deploying daemon node-exporter.b on smithi129 2023-07-29T17:19:45.164 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:44 smithi129 bash[53756]: cluster 2023-07-29T17:19:43.890060+0000 mgr.y (mgr.25169) 411 : cluster 0 pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:45.164 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:44 smithi129 bash[69947]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-node-exporter.b 2023-07-29T17:19:45.164 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:44 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:19:45.164 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:44 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-29T17:19:45.164 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:44 smithi129 systemd[1]: Stopped Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:19:45.165 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:45 smithi129 systemd[1]: Started Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:19:45.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[60360]: cephadm 2023-07-29T17:19:43.332618+0000 mgr.y (mgr.25169) 409 : cephadm 1 Upgrade: Updating node-exporter.b (2/2) 2023-07-29T17:19:45.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[60360]: cephadm 2023-07-29T17:19:43.333363+0000 mgr.y (mgr.25169) 410 : cephadm 1 Deploying daemon node-exporter.b on smithi129 2023-07-29T17:19:45.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[60360]: cluster 2023-07-29T17:19:43.890060+0000 mgr.y (mgr.25169) 411 : cluster 0 pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:45.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[58614]: cephadm 2023-07-29T17:19:43.332618+0000 mgr.y (mgr.25169) 409 : cephadm 1 Upgrade: Updating node-exporter.b (2/2) 2023-07-29T17:19:45.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[58614]: cephadm 2023-07-29T17:19:43.333363+0000 mgr.y (mgr.25169) 410 : cephadm 1 Deploying daemon node-exporter.b on smithi129 2023-07-29T17:19:45.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[58614]: cluster 2023-07-29T17:19:43.890060+0000 mgr.y (mgr.25169) 411 : cluster 0 pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 222 B/s wr, 4 op/s 2023-07-29T17:19:45.196 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.854Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2023-07-29T17:19:45.196 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.854Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2023-07-29T17:19:45.196 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.855Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2023-07-29T17:19:45.196 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.855Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2023-07-29T17:19:45.197 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2023-07-29T17:19:45.197 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-07-29T17:19:45.197 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2023-07-29T17:19:45.197 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=arp 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=bcache 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=bonding 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=btrfs 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=conntrack 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=cpu 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=cpufreq 2023-07-29T17:19:45.198 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=diskstats 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=dmi 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=edac 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=entropy 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=fibrechannel 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=filefd 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=filesystem 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=hwmon 2023-07-29T17:19:45.199 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=infiniband 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=ipvs 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=loadavg 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=mdadm 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=meminfo 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=netclass 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=netdev 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=netstat 2023-07-29T17:19:45.200 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=nfs 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=nfsd 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=nvme 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=os 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=pressure 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=rapl 2023-07-29T17:19:45.201 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=schedstat 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=selinux 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=sockstat 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=softnet 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=stat 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=tapestats 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=textfile 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-07-29T17:19:45.202 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=time 2023-07-29T17:19:45.203 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-07-29T17:19:45.203 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=uname 2023-07-29T17:19:45.203 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=vmstat 2023-07-29T17:19:45.203 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=xfs 2023-07-29T17:19:45.203 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.856Z caller=node_exporter.go:117 level=info collector=zfs 2023-07-29T17:19:45.203 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.857Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2023-07-29T17:19:45.204 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:19:44 smithi078 bash[77020]: ts=2023-07-29T17:19:44.857Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2023-07-29T17:19:45.426 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:45 smithi129 bash[70111]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2023-07-29T17:19:46.091 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:45 smithi129 bash[70111]: v1.5.0: Pulling from prometheus/node-exporter 2023-07-29T17:19:46.362 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 22b70bddd3ac: Pulling fs layer 2023-07-29T17:19:46.362 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 5c12815fee55: Pulling fs layer 2023-07-29T17:19:46.363 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: c0e87333d380: Pulling fs layer 2023-07-29T17:19:46.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[53756]: audit 2023-07-29T17:19:45.180873+0000 mon.a (mon.0) 864 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[53756]: audit 2023-07-29T17:19:45.190338+0000 mon.a (mon.0) 865 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[53756]: audit 2023-07-29T17:19:45.579195+0000 mon.a (mon.0) 866 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[53756]: audit 2023-07-29T17:19:45.592899+0000 mon.a (mon.0) 867 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.627 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 5c12815fee55: Verifying Checksum 2023-07-29T17:19:46.627 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 5c12815fee55: Download complete 2023-07-29T17:19:46.627 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 22b70bddd3ac: Download complete 2023-07-29T17:19:46.628 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: c0e87333d380: Verifying Checksum 2023-07-29T17:19:46.628 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: c0e87333d380: Download complete 2023-07-29T17:19:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[58614]: audit 2023-07-29T17:19:45.180873+0000 mon.a (mon.0) 864 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[58614]: audit 2023-07-29T17:19:45.190338+0000 mon.a (mon.0) 865 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[58614]: audit 2023-07-29T17:19:45.579195+0000 mon.a (mon.0) 866 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[58614]: audit 2023-07-29T17:19:45.592899+0000 mon.a (mon.0) 867 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[60360]: audit 2023-07-29T17:19:45.180873+0000 mon.a (mon.0) 864 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[60360]: audit 2023-07-29T17:19:45.190338+0000 mon.a (mon.0) 865 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[60360]: audit 2023-07-29T17:19:45.579195+0000 mon.a (mon.0) 866 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:46 smithi078 bash[60360]: audit 2023-07-29T17:19:45.592899+0000 mon.a (mon.0) 867 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:46.925 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 22b70bddd3ac: Pull complete 2023-07-29T17:19:47.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:47 smithi129 bash[53756]: cluster 2023-07-29T17:19:45.891176+0000 mgr.y (mgr.25169) 412 : cluster 0 pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 204 B/s wr, 3 op/s 2023-07-29T17:19:47.383 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:46 smithi129 bash[70111]: 5c12815fee55: Pull complete 2023-07-29T17:19:47.633 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:47 smithi129 bash[70111]: c0e87333d380: Pull complete 2023-07-29T17:19:47.633 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:47 smithi129 bash[70111]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2023-07-29T17:19:47.634 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:47 smithi129 bash[70111]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2023-07-29T17:19:47.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:47 smithi078 bash[58614]: cluster 2023-07-29T17:19:45.891176+0000 mgr.y (mgr.25169) 412 : cluster 0 pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 204 B/s wr, 3 op/s 2023-07-29T17:19:47.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:47 smithi078 bash[60360]: cluster 2023-07-29T17:19:45.891176+0000 mgr.y (mgr.25169) 412 : cluster 0 pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 204 B/s wr, 3 op/s 2023-07-29T17:19:49.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[53756]: cluster 2023-07-29T17:19:47.892491+0000 mgr.y (mgr.25169) 413 : cluster 0 pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 170 B/s wr, 3 op/s 2023-07-29T17:19:49.426 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.189Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2023-07-29T17:19:49.426 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.189Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.190Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.190Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=arp 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=bcache 2023-07-29T17:19:49.427 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=bonding 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=btrfs 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=conntrack 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=cpu 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=cpufreq 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=diskstats 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=dmi 2023-07-29T17:19:49.428 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=edac 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=entropy 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=fibrechannel 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=filefd 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=filesystem 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=hwmon 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=infiniband 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=ipvs 2023-07-29T17:19:49.429 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=loadavg 2023-07-29T17:19:49.430 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=mdadm 2023-07-29T17:19:49.430 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=meminfo 2023-07-29T17:19:49.430 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=netclass 2023-07-29T17:19:49.430 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=netdev 2023-07-29T17:19:49.430 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=netstat 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=nfs 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=nfsd 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=nvme 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=os 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=pressure 2023-07-29T17:19:49.431 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=rapl 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=schedstat 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=selinux 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=sockstat 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=softnet 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=stat 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=tapestats 2023-07-29T17:19:49.432 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=textfile 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=time 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.191Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.192Z caller=node_exporter.go:117 level=info collector=uname 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.192Z caller=node_exporter.go:117 level=info collector=vmstat 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.192Z caller=node_exporter.go:117 level=info collector=xfs 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.192Z caller=node_exporter.go:117 level=info collector=zfs 2023-07-29T17:19:49.433 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.192Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2023-07-29T17:19:49.434 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:19:49 smithi129 bash[70111]: ts=2023-07-29T17:19:49.192Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2023-07-29T17:19:49.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:49 smithi078 bash[58614]: cluster 2023-07-29T17:19:47.892491+0000 mgr.y (mgr.25169) 413 : cluster 0 pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 170 B/s wr, 3 op/s 2023-07-29T17:19:49.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:49 smithi078 bash[60360]: cluster 2023-07-29T17:19:47.892491+0000 mgr.y (mgr.25169) 413 : cluster 0 pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 170 B/s wr, 3 op/s 2023-07-29T17:19:51.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:51 smithi129 bash[53756]: audit 2023-07-29T17:19:49.862611+0000 mgr.y (mgr.25169) 414 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:51.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:51 smithi129 bash[53756]: cluster 2023-07-29T17:19:49.893954+0000 mgr.y (mgr.25169) 415 : cluster 0 pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 2 op/s 2023-07-29T17:19:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:51 smithi078 bash[60360]: audit 2023-07-29T17:19:49.862611+0000 mgr.y (mgr.25169) 414 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:51.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:51 smithi078 bash[60360]: cluster 2023-07-29T17:19:49.893954+0000 mgr.y (mgr.25169) 415 : cluster 0 pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 2 op/s 2023-07-29T17:19:51.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:51 smithi078 bash[58614]: audit 2023-07-29T17:19:49.862611+0000 mgr.y (mgr.25169) 414 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:19:51.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:51 smithi078 bash[58614]: cluster 2023-07-29T17:19:49.893954+0000 mgr.y (mgr.25169) 415 : cluster 0 pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 2 op/s 2023-07-29T17:19:52.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:52 smithi129 bash[53756]: audit 2023-07-29T17:19:51.247232+0000 mon.a (mon.0) 868 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:52 smithi129 bash[53756]: audit 2023-07-29T17:19:51.259055+0000 mon.a (mon.0) 869 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:52 smithi129 bash[53756]: cluster 2023-07-29T17:19:51.894992+0000 mgr.y (mgr.25169) 416 : cluster 0 pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:52.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:52 smithi129 bash[53756]: audit 2023-07-29T17:19:52.055503+0000 mon.a (mon.0) 870 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:52 smithi129 bash[53756]: audit 2023-07-29T17:19:52.067235+0000 mon.a (mon.0) 871 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[60360]: audit 2023-07-29T17:19:51.247232+0000 mon.a (mon.0) 868 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[60360]: audit 2023-07-29T17:19:51.259055+0000 mon.a (mon.0) 869 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[60360]: cluster 2023-07-29T17:19:51.894992+0000 mgr.y (mgr.25169) 416 : cluster 0 pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:52.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[60360]: audit 2023-07-29T17:19:52.055503+0000 mon.a (mon.0) 870 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[60360]: audit 2023-07-29T17:19:52.067235+0000 mon.a (mon.0) 871 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[58614]: audit 2023-07-29T17:19:51.247232+0000 mon.a (mon.0) 868 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[58614]: audit 2023-07-29T17:19:51.259055+0000 mon.a (mon.0) 869 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[58614]: cluster 2023-07-29T17:19:51.894992+0000 mgr.y (mgr.25169) 416 : cluster 0 pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:52.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[58614]: audit 2023-07-29T17:19:52.055503+0000 mon.a (mon.0) 870 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:52.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:52 smithi078 bash[58614]: audit 2023-07-29T17:19:52.067235+0000 mon.a (mon.0) 871 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:54 smithi129 bash[53756]: audit 2023-07-29T17:19:53.011353+0000 mon.a (mon.0) 872 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:54 smithi129 bash[53756]: audit 2023-07-29T17:19:53.020835+0000 mon.a (mon.0) 873 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:54 smithi129 bash[53756]: audit 2023-07-29T17:19:53.822987+0000 mon.a (mon.0) 874 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:54 smithi129 bash[53756]: audit 2023-07-29T17:19:53.834034+0000 mon.a (mon.0) 875 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[58614]: audit 2023-07-29T17:19:53.011353+0000 mon.a (mon.0) 872 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[58614]: audit 2023-07-29T17:19:53.020835+0000 mon.a (mon.0) 873 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[58614]: audit 2023-07-29T17:19:53.822987+0000 mon.a (mon.0) 874 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[58614]: audit 2023-07-29T17:19:53.834034+0000 mon.a (mon.0) 875 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[60360]: audit 2023-07-29T17:19:53.011353+0000 mon.a (mon.0) 872 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[60360]: audit 2023-07-29T17:19:53.020835+0000 mon.a (mon.0) 873 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[60360]: audit 2023-07-29T17:19:53.822987+0000 mon.a (mon.0) 874 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:54.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:54 smithi078 bash[60360]: audit 2023-07-29T17:19:53.834034+0000 mon.a (mon.0) 875 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:19:55.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:55 smithi129 bash[53756]: cluster 2023-07-29T17:19:53.896401+0000 mgr.y (mgr.25169) 417 : cluster 0 pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:19:55.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:55 smithi078 bash[58614]: cluster 2023-07-29T17:19:53.896401+0000 mgr.y (mgr.25169) 417 : cluster 0 pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:19:55.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:55 smithi078 bash[60360]: cluster 2023-07-29T17:19:53.896401+0000 mgr.y (mgr.25169) 417 : cluster 0 pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:19:57.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:57 smithi129 bash[53756]: cluster 2023-07-29T17:19:55.897509+0000 mgr.y (mgr.25169) 418 : cluster 0 pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:57.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:57 smithi078 bash[58614]: cluster 2023-07-29T17:19:55.897509+0000 mgr.y (mgr.25169) 418 : cluster 0 pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:57.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:57 smithi078 bash[60360]: cluster 2023-07-29T17:19:55.897509+0000 mgr.y (mgr.25169) 418 : cluster 0 pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:59.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:19:59 smithi129 bash[53756]: cluster 2023-07-29T17:19:57.898780+0000 mgr.y (mgr.25169) 419 : cluster 0 pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:59.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:19:59 smithi078 bash[58614]: cluster 2023-07-29T17:19:57.898780+0000 mgr.y (mgr.25169) 419 : cluster 0 pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:19:59.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:19:59 smithi078 bash[60360]: cluster 2023-07-29T17:19:57.898780+0000 mgr.y (mgr.25169) 419 : cluster 0 pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:00.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:00 smithi129 bash[53756]: cluster 2023-07-29T17:20:00.000196+0000 mon.a (mon.0) 876 : cluster 3 Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-29T17:20:00.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:00 smithi129 bash[53756]: cluster 2023-07-29T17:20:00.000285+0000 mon.a (mon.0) 877 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-29T17:20:00.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:00 smithi129 bash[53756]: cluster 2023-07-29T17:20:00.000325+0000 mon.a (mon.0) 878 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:20:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:00 smithi078 bash[60360]: cluster 2023-07-29T17:20:00.000196+0000 mon.a (mon.0) 876 : cluster 3 Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-29T17:20:00.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:00 smithi078 bash[60360]: cluster 2023-07-29T17:20:00.000285+0000 mon.a (mon.0) 877 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-29T17:20:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:00 smithi078 bash[60360]: cluster 2023-07-29T17:20:00.000325+0000 mon.a (mon.0) 878 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:20:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:00 smithi078 bash[58614]: cluster 2023-07-29T17:20:00.000196+0000 mon.a (mon.0) 876 : cluster 3 Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-29T17:20:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:00 smithi078 bash[58614]: cluster 2023-07-29T17:20:00.000285+0000 mon.a (mon.0) 877 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-29T17:20:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:00 smithi078 bash[58614]: cluster 2023-07-29T17:20:00.000325+0000 mon.a (mon.0) 878 : cluster 3 daemon prometheus.a on smithi129 is in error state 2023-07-29T17:20:01.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:01 smithi129 bash[53756]: audit 2023-07-29T17:19:59.871157+0000 mgr.y (mgr.25169) 420 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:01.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:01 smithi129 bash[53756]: cluster 2023-07-29T17:19:59.900159+0000 mgr.y (mgr.25169) 421 : cluster 0 pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:01 smithi078 bash[58614]: audit 2023-07-29T17:19:59.871157+0000 mgr.y (mgr.25169) 420 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:01 smithi078 bash[58614]: cluster 2023-07-29T17:19:59.900159+0000 mgr.y (mgr.25169) 421 : cluster 0 pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:01.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:01 smithi078 bash[60360]: audit 2023-07-29T17:19:59.871157+0000 mgr.y (mgr.25169) 420 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:01.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:01 smithi078 bash[60360]: cluster 2023-07-29T17:19:59.900159+0000 mgr.y (mgr.25169) 421 : cluster 0 pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:03.129 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:02 smithi129 bash[53756]: audit 2023-07-29T17:20:01.827275+0000 mon.a (mon.0) 879 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:03.129 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:02 smithi129 bash[53756]: audit 2023-07-29T17:20:01.838590+0000 mon.a (mon.0) 880 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:03.129 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:02 smithi129 bash[53756]: cluster 2023-07-29T17:20:01.901254+0000 mgr.y (mgr.25169) 422 : cluster 0 pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:02 smithi078 bash[58614]: audit 2023-07-29T17:20:01.827275+0000 mon.a (mon.0) 879 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:02 smithi078 bash[58614]: audit 2023-07-29T17:20:01.838590+0000 mon.a (mon.0) 880 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:03.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:02 smithi078 bash[58614]: cluster 2023-07-29T17:20:01.901254+0000 mgr.y (mgr.25169) 422 : cluster 0 pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:03.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:02 smithi078 bash[60360]: audit 2023-07-29T17:20:01.827275+0000 mon.a (mon.0) 879 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:02 smithi078 bash[60360]: audit 2023-07-29T17:20:01.838590+0000 mon.a (mon.0) 880 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:02 smithi078 bash[60360]: cluster 2023-07-29T17:20:01.901254+0000 mgr.y (mgr.25169) 422 : cluster 0 pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:05.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.735488+0000 mon.a (mon.0) 881 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.748004+0000 mon.a (mon.0) 882 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.749673+0000 mon.a (mon.0) 883 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.751677+0000 mon.a (mon.0) 884 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: cephadm 2023-07-29T17:20:03.753176+0000 mgr.y (mgr.25169) 423 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: cluster 2023-07-29T17:20:03.754864+0000 mgr.y (mgr.25169) 424 : cluster 0 pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: cluster 2023-07-29T17:20:03.755336+0000 mgr.y (mgr.25169) 425 : cluster 0 pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.764603+0000 mon.a (mon.0) 885 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.852247+0000 mon.a (mon.0) 886 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.857452+0000 mon.a (mon.0) 887 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.860111+0000 mon.a (mon.0) 888 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.862365+0000 mon.a (mon.0) 889 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.865864+0000 mon.a (mon.0) 890 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.869439+0000 mon.a (mon.0) 891 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.178 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.872338+0000 mon.a (mon.0) 892 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.874422+0000 mon.a (mon.0) 893 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.876571+0000 mon.a (mon.0) 894 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.878888+0000 mon.a (mon.0) 895 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.881268+0000 mon.a (mon.0) 896 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.883667+0000 mon.a (mon.0) 897 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.179 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:04 smithi129 bash[53756]: audit 2023-07-29T17:20:03.886778+0000 mon.a (mon.0) 898 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.735488+0000 mon.a (mon.0) 881 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.748004+0000 mon.a (mon.0) 882 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.749673+0000 mon.a (mon.0) 883 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.751677+0000 mon.a (mon.0) 884 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: cephadm 2023-07-29T17:20:03.753176+0000 mgr.y (mgr.25169) 423 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: cluster 2023-07-29T17:20:03.754864+0000 mgr.y (mgr.25169) 424 : cluster 0 pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: cluster 2023-07-29T17:20:03.755336+0000 mgr.y (mgr.25169) 425 : cluster 0 pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.764603+0000 mon.a (mon.0) 885 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.852247+0000 mon.a (mon.0) 886 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.857452+0000 mon.a (mon.0) 887 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.860111+0000 mon.a (mon.0) 888 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.862365+0000 mon.a (mon.0) 889 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.865864+0000 mon.a (mon.0) 890 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.869439+0000 mon.a (mon.0) 891 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.872338+0000 mon.a (mon.0) 892 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.874422+0000 mon.a (mon.0) 893 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.876571+0000 mon.a (mon.0) 894 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.878888+0000 mon.a (mon.0) 895 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.881268+0000 mon.a (mon.0) 896 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.883667+0000 mon.a (mon.0) 897 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[58614]: audit 2023-07-29T17:20:03.886778+0000 mon.a (mon.0) 898 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.735488+0000 mon.a (mon.0) 881 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.748004+0000 mon.a (mon.0) 882 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.749673+0000 mon.a (mon.0) 883 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.751677+0000 mon.a (mon.0) 884 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: cephadm 2023-07-29T17:20:03.753176+0000 mgr.y (mgr.25169) 423 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: cluster 2023-07-29T17:20:03.754864+0000 mgr.y (mgr.25169) 424 : cluster 0 pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: cluster 2023-07-29T17:20:03.755336+0000 mgr.y (mgr.25169) 425 : cluster 0 pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.764603+0000 mon.a (mon.0) 885 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:05.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.852247+0000 mon.a (mon.0) 886 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.857452+0000 mon.a (mon.0) 887 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.860111+0000 mon.a (mon.0) 888 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.862365+0000 mon.a (mon.0) 889 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.865864+0000 mon.a (mon.0) 890 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.869439+0000 mon.a (mon.0) 891 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.872338+0000 mon.a (mon.0) 892 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.874422+0000 mon.a (mon.0) 893 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.876571+0000 mon.a (mon.0) 894 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.878888+0000 mon.a (mon.0) 895 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.881268+0000 mon.a (mon.0) 896 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.883667+0000 mon.a (mon.0) 897 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:05.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:04 smithi078 bash[60360]: audit 2023-07-29T17:20:03.886778+0000 mon.a (mon.0) 898 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:06 smithi129 bash[53756]: cluster 2023-07-29T17:20:05.756382+0000 mgr.y (mgr.25169) 426 : cluster 0 pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:07.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:06 smithi129 bash[53756]: cephadm 2023-07-29T17:20:06.159377+0000 mgr.y (mgr.25169) 427 : cephadm 1 Upgrade: Updating prometheus.a 2023-07-29T17:20:07.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:06 smithi078 bash[58614]: cluster 2023-07-29T17:20:05.756382+0000 mgr.y (mgr.25169) 426 : cluster 0 pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:07.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:06 smithi078 bash[58614]: cephadm 2023-07-29T17:20:06.159377+0000 mgr.y (mgr.25169) 427 : cephadm 1 Upgrade: Updating prometheus.a 2023-07-29T17:20:07.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:06 smithi078 bash[60360]: cluster 2023-07-29T17:20:05.756382+0000 mgr.y (mgr.25169) 426 : cluster 0 pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:07.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:06 smithi078 bash[60360]: cephadm 2023-07-29T17:20:06.159377+0000 mgr.y (mgr.25169) 427 : cephadm 1 Upgrade: Updating prometheus.a 2023-07-29T17:20:08.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:07 smithi129 bash[53756]: cephadm 2023-07-29T17:20:06.404318+0000 mgr.y (mgr.25169) 428 : cephadm 1 Deploying daemon prometheus.a on smithi129 2023-07-29T17:20:08.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:07 smithi078 bash[58614]: cephadm 2023-07-29T17:20:06.404318+0000 mgr.y (mgr.25169) 428 : cephadm 1 Deploying daemon prometheus.a on smithi129 2023-07-29T17:20:08.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:07 smithi078 bash[60360]: cephadm 2023-07-29T17:20:06.404318+0000 mgr.y (mgr.25169) 428 : cephadm 1 Deploying daemon prometheus.a on smithi129 2023-07-29T17:20:09.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:08 smithi129 bash[53756]: cluster 2023-07-29T17:20:07.757622+0000 mgr.y (mgr.25169) 429 : cluster 0 pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:09.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:08 smithi078 bash[60360]: cluster 2023-07-29T17:20:07.757622+0000 mgr.y (mgr.25169) 429 : cluster 0 pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:09.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:08 smithi078 bash[58614]: cluster 2023-07-29T17:20:07.757622+0000 mgr.y (mgr.25169) 429 : cluster 0 pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:09.294 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:20:10.039 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:20:10.039 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (10m) 18s ago 16m 11.0M - 0.20.0 0881eb8f169f 13b0be387d50 2023-07-29T17:20:10.040 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (9m) 17s ago 16m 26.4M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:20:10.040 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (52s) 17s ago 13m 46.3M - 3.6 b662b4d80c86 a64a43bc0b16 2023-07-29T17:20:10.040 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (8m) 17s ago 20m 430M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:20:10.041 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (9m) 18s ago 22m 480M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:20:10.041 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7m) 18s ago 22m 82.1M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:20:10.041 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (6m) 17s ago 21m 61.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:20:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7m) 18s ago 20m 61.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:20:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (26s) 18s ago 16m 3840k - 1.5.0 0da6a335fe13 4196bcd9b292 2023-07-29T17:20:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (22s) 17s ago 16m - - 1.5.0 0da6a335fe13 f9ecb2f5f5ff 2023-07-29T17:20:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5m) 18s ago 20m 62.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:20:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4m) 18s ago 19m 65.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:20:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4m) 18s ago 19m 54.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:20:10.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3m) 18s ago 18m 61.4M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:20:10.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (3m) 17s ago 18m 61.6M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:20:10.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (2m) 17s ago 18m 55.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:20:10.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (2m) 17s ago 17m 54.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:20:10.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (118s) 17s ago 17m 56.1M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:20:10.045 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 error 17s ago 16m - - 2023-07-29T17:20:10.045 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (94s) 18s ago 14m 84.0M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:20:10.045 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (81s) 17s ago 14m 84.0M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:20:10.045 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (87s) 18s ago 14m 84.2M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:20:10.046 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (74s) 17s ago 14m 84.0M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:20:10.477 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:20:10.477 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:20:10.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:20:10.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:10.478 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:20:10.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:20:10.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:10.478 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:20:10.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:20:10.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:20:10.480 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:20:10.854 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:20:10.855 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:20:10.855 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:20:10.855 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:20:10.855 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:20:10.855 INFO:teuthology.orchestra.run.smithi078.stdout: "osd", 2023-07-29T17:20:10.855 INFO:teuthology.orchestra.run.smithi078.stdout: "iscsi", 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "18/23 daemons upgraded", 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading prometheus daemons", 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:20:10.856 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:20:11.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:10 smithi129 bash[53756]: audit 2023-07-29T17:20:09.255716+0000 mgr.y (mgr.25169) 430 : audit 0 from='client.34935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:10 smithi129 bash[53756]: audit 2023-07-29T17:20:09.628361+0000 mgr.y (mgr.25169) 431 : audit 0 from='client.44783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:10 smithi129 bash[53756]: cluster 2023-07-29T17:20:09.758450+0000 mgr.y (mgr.25169) 432 : cluster 0 pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:11.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:10 smithi129 bash[53756]: audit 2023-07-29T17:20:09.876790+0000 mgr.y (mgr.25169) 433 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:11.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:10 smithi129 bash[53756]: audit 2023-07-29T17:20:09.992423+0000 mgr.y (mgr.25169) 434 : audit 0 from='client.44789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:10 smithi129 bash[53756]: audit 2023-07-29T17:20:10.452731+0000 mon.c (mon.2) 16 : audit 0 from='client.34953 172.21.15.78:0/1847626444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:11.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[58614]: audit 2023-07-29T17:20:09.255716+0000 mgr.y (mgr.25169) 430 : audit 0 from='client.34935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[58614]: audit 2023-07-29T17:20:09.628361+0000 mgr.y (mgr.25169) 431 : audit 0 from='client.44783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[58614]: cluster 2023-07-29T17:20:09.758450+0000 mgr.y (mgr.25169) 432 : cluster 0 pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:11.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[58614]: audit 2023-07-29T17:20:09.876790+0000 mgr.y (mgr.25169) 433 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:11.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[58614]: audit 2023-07-29T17:20:09.992423+0000 mgr.y (mgr.25169) 434 : audit 0 from='client.44789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[58614]: audit 2023-07-29T17:20:10.452731+0000 mon.c (mon.2) 16 : audit 0 from='client.34953 172.21.15.78:0/1847626444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[60360]: audit 2023-07-29T17:20:09.255716+0000 mgr.y (mgr.25169) 430 : audit 0 from='client.34935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[60360]: audit 2023-07-29T17:20:09.628361+0000 mgr.y (mgr.25169) 431 : audit 0 from='client.44783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[60360]: cluster 2023-07-29T17:20:09.758450+0000 mgr.y (mgr.25169) 432 : cluster 0 pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[60360]: audit 2023-07-29T17:20:09.876790+0000 mgr.y (mgr.25169) 433 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[60360]: audit 2023-07-29T17:20:09.992423+0000 mgr.y (mgr.25169) 434 : audit 0 from='client.44789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:11.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:10 smithi078 bash[60360]: audit 2023-07-29T17:20:10.452731+0000 mon.c (mon.2) 16 : audit 0 from='client.34953 172.21.15.78:0/1847626444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:11.288 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-29T17:20:11.288 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-29T17:20:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi129 is in error state 2023-07-29T17:20:11.676 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:11 smithi129 systemd[1]: Started Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:20:12.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:11 smithi129 bash[53756]: audit 2023-07-29T17:20:10.829320+0000 mgr.y (mgr.25169) 435 : audit 0 from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:12.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:11 smithi129 bash[53756]: audit 2023-07-29T17:20:11.266201+0000 mon.b (mon.1) 18 : audit 0 from='client.34965 172.21.15.78:0/1185759385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:20:12.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:11 smithi129 bash[53756]: audit 2023-07-29T17:20:11.292247+0000 mon.a (mon.0) 899 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:12.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:11 smithi129 bash[53756]: audit 2023-07-29T17:20:11.301189+0000 mon.a (mon.0) 900 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[58614]: audit 2023-07-29T17:20:10.829320+0000 mgr.y (mgr.25169) 435 : audit 0 from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[58614]: audit 2023-07-29T17:20:11.266201+0000 mon.b (mon.1) 18 : audit 0 from='client.34965 172.21.15.78:0/1185759385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:20:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[58614]: audit 2023-07-29T17:20:11.292247+0000 mon.a (mon.0) 899 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[58614]: audit 2023-07-29T17:20:11.301189+0000 mon.a (mon.0) 900 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:12.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[60360]: audit 2023-07-29T17:20:10.829320+0000 mgr.y (mgr.25169) 435 : audit 0 from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:12.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[60360]: audit 2023-07-29T17:20:11.266201+0000 mon.b (mon.1) 18 : audit 0 from='client.34965 172.21.15.78:0/1185759385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:20:12.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[60360]: audit 2023-07-29T17:20:11.292247+0000 mon.a (mon.0) 899 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:12.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:11 smithi078 bash[60360]: audit 2023-07-29T17:20:11.301189+0000 mon.a (mon.0) 900 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:12.859 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.650Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2023-07-29T17:20:12.859 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.650Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2023-07-29T17:20:12.860 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.650Z caller=main.go:570 level=info host_details="(Linux 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 smithi129 (none))" 2023-07-29T17:20:12.860 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.650Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-07-29T17:20:12.860 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.650Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-29T17:20:12.860 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.652Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2023-07-29T17:20:12.861 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.652Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2023-07-29T17:20:12.861 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.653Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock 2023-07-29T17:20:12.861 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.653Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2023-07-29T17:20:12.862 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.653Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2023-07-29T17:20:12.862 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.658Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-07-29T17:20:12.862 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.658Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.192µs 2023-07-29T17:20:12.862 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.658Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-07-29T17:20:12.863 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.659Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=13 2023-07-29T17:20:12.863 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.673Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=13 2023-07-29T17:20:12.864 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.674Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=13 2023-07-29T17:20:12.864 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.674Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=13 2023-07-29T17:20:12.864 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.674Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=13 2023-07-29T17:20:12.864 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.675Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=13 2023-07-29T17:20:12.865 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.675Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=13 2023-07-29T17:20:12.865 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.675Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=13 2023-07-29T17:20:12.865 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.675Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=13 2023-07-29T17:20:12.865 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.676Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=13 2023-07-29T17:20:12.865 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.676Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=10 maxSegment=13 2023-07-29T17:20:12.866 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.676Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=11 maxSegment=13 2023-07-29T17:20:12.866 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.676Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=12 maxSegment=13 2023-07-29T17:20:12.866 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.677Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=13 maxSegment=13 2023-07-29T17:20:12.866 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.677Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=74.134µs wal_replay_duration=18.107616ms wbl_replay_duration=116ns total_replay_duration=18.216215ms 2023-07-29T17:20:12.867 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.678Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2023-07-29T17:20:12.867 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.678Z caller=main.go:1029 level=info msg="TSDB started" 2023-07-29T17:20:12.867 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.678Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-29T17:20:12.867 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.702Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.661734ms db_storage=1.173µs remote_storage=1.46µs web_handler=463ns query_engine=919ns scrape=349.217µs scrape_sd=111.366µs notify=20.821µs notify_sd=20.206µs rules=22.847346ms tracing=8.155µs 2023-07-29T17:20:12.868 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.702Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2023-07-29T17:20:12.868 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[71906]: ts=2023-07-29T17:20:12.702Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2023-07-29T17:20:13.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:12 smithi129 bash[53756]: cluster 2023-07-29T17:20:11.759463+0000 mgr.y (mgr.25169) 436 : cluster 0 pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:13.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:12 smithi078 bash[58614]: cluster 2023-07-29T17:20:11.759463+0000 mgr.y (mgr.25169) 436 : cluster 0 pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:13.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:12 smithi078 bash[60360]: cluster 2023-07-29T17:20:11.759463+0000 mgr.y (mgr.25169) 436 : cluster 0 pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-07-29T17:20:15.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:14 smithi129 bash[53756]: cluster 2023-07-29T17:20:13.761057+0000 mgr.y (mgr.25169) 437 : cluster 0 pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-29T17:20:15.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:14 smithi078 bash[58614]: cluster 2023-07-29T17:20:13.761057+0000 mgr.y (mgr.25169) 437 : cluster 0 pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-29T17:20:15.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:14 smithi078 bash[60360]: cluster 2023-07-29T17:20:13.761057+0000 mgr.y (mgr.25169) 437 : cluster 0 pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-29T17:20:17.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:16 smithi129 bash[53756]: cluster 2023-07-29T17:20:15.762125+0000 mgr.y (mgr.25169) 438 : cluster 0 pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:17.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:16 smithi078 bash[58614]: cluster 2023-07-29T17:20:15.762125+0000 mgr.y (mgr.25169) 438 : cluster 0 pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:17.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:16 smithi078 bash[60360]: cluster 2023-07-29T17:20:15.762125+0000 mgr.y (mgr.25169) 438 : cluster 0 pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:18 smithi129 bash[53756]: cluster 2023-07-29T17:20:17.763278+0000 mgr.y (mgr.25169) 439 : cluster 0 pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:18 smithi129 bash[53756]: audit 2023-07-29T17:20:18.782191+0000 mon.a (mon.0) 901 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:19.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:18 smithi129 bash[53756]: audit 2023-07-29T17:20:18.792311+0000 mon.a (mon.0) 902 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:19.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:18 smithi078 bash[58614]: cluster 2023-07-29T17:20:17.763278+0000 mgr.y (mgr.25169) 439 : cluster 0 pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:19.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:18 smithi078 bash[58614]: audit 2023-07-29T17:20:18.782191+0000 mon.a (mon.0) 901 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:19.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:18 smithi078 bash[58614]: audit 2023-07-29T17:20:18.792311+0000 mon.a (mon.0) 902 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:18 smithi078 bash[60360]: cluster 2023-07-29T17:20:17.763278+0000 mgr.y (mgr.25169) 439 : cluster 0 pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:18 smithi078 bash[60360]: audit 2023-07-29T17:20:18.782191+0000 mon.a (mon.0) 901 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:19.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:18 smithi078 bash[60360]: audit 2023-07-29T17:20:18.792311+0000 mon.a (mon.0) 902 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:20 smithi129 bash[53756]: audit 2023-07-29T17:20:19.594601+0000 mon.a (mon.0) 903 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:20 smithi129 bash[53756]: audit 2023-07-29T17:20:19.605282+0000 mon.a (mon.0) 904 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:20 smithi129 bash[53756]: cluster 2023-07-29T17:20:19.764607+0000 mgr.y (mgr.25169) 440 : cluster 0 pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:20.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:20 smithi129 bash[53756]: audit 2023-07-29T17:20:19.885249+0000 mgr.y (mgr.25169) 441 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:20.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[60360]: audit 2023-07-29T17:20:19.594601+0000 mon.a (mon.0) 903 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[60360]: audit 2023-07-29T17:20:19.605282+0000 mon.a (mon.0) 904 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[60360]: cluster 2023-07-29T17:20:19.764607+0000 mgr.y (mgr.25169) 440 : cluster 0 pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:20.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[60360]: audit 2023-07-29T17:20:19.885249+0000 mgr.y (mgr.25169) 441 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[58614]: audit 2023-07-29T17:20:19.594601+0000 mon.a (mon.0) 903 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[58614]: audit 2023-07-29T17:20:19.605282+0000 mon.a (mon.0) 904 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[58614]: cluster 2023-07-29T17:20:19.764607+0000 mgr.y (mgr.25169) 440 : cluster 0 pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:20.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:20 smithi078 bash[58614]: audit 2023-07-29T17:20:19.885249+0000 mgr.y (mgr.25169) 441 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:23.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:22 smithi129 bash[53756]: cluster 2023-07-29T17:20:21.765566+0000 mgr.y (mgr.25169) 442 : cluster 0 pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:23.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:22 smithi078 bash[58614]: cluster 2023-07-29T17:20:21.765566+0000 mgr.y (mgr.25169) 442 : cluster 0 pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:23.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:22 smithi078 bash[60360]: cluster 2023-07-29T17:20:21.765566+0000 mgr.y (mgr.25169) 442 : cluster 0 pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:25.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:24 smithi129 bash[53756]: cluster 2023-07-29T17:20:23.767101+0000 mgr.y (mgr.25169) 443 : cluster 0 pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:25.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:24 smithi078 bash[58614]: cluster 2023-07-29T17:20:23.767101+0000 mgr.y (mgr.25169) 443 : cluster 0 pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:25.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:24 smithi078 bash[60360]: cluster 2023-07-29T17:20:23.767101+0000 mgr.y (mgr.25169) 443 : cluster 0 pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:27.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:26 smithi129 bash[53756]: cluster 2023-07-29T17:20:25.768169+0000 mgr.y (mgr.25169) 444 : cluster 0 pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:27.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:26 smithi078 bash[60360]: cluster 2023-07-29T17:20:25.768169+0000 mgr.y (mgr.25169) 444 : cluster 0 pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:27.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:26 smithi078 bash[58614]: cluster 2023-07-29T17:20:25.768169+0000 mgr.y (mgr.25169) 444 : cluster 0 pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:29.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:28 smithi129 bash[53756]: cluster 2023-07-29T17:20:27.769314+0000 mgr.y (mgr.25169) 445 : cluster 0 pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:29.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:28 smithi078 bash[58614]: cluster 2023-07-29T17:20:27.769314+0000 mgr.y (mgr.25169) 445 : cluster 0 pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:29.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:28 smithi078 bash[60360]: cluster 2023-07-29T17:20:27.769314+0000 mgr.y (mgr.25169) 445 : cluster 0 pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:30.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.560290+0000 mon.a (mon.0) 905 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.569852+0000 mon.a (mon.0) 906 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.571621+0000 mon.a (mon.0) 907 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:30.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.572850+0000 mon.a (mon.0) 908 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:30.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: cephadm 2023-07-29T17:20:29.573960+0000 mgr.y (mgr.25169) 446 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: cluster 2023-07-29T17:20:29.575578+0000 mgr.y (mgr.25169) 447 : cluster 0 pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.581542+0000 mon.a (mon.0) 909 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.662940+0000 mon.a (mon.0) 910 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.664036+0000 mgr.y (mgr.25169) 448 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.673269+0000 mon.a (mon.0) 911 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.678210+0000 mon.a (mon.0) 912 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.680855+0000 mon.a (mon.0) 913 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.682880+0000 mon.a (mon.0) 914 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.686429+0000 mon.a (mon.0) 915 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.690411+0000 mon.a (mon.0) 916 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.693437+0000 mon.a (mon.0) 917 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.695721+0000 mon.a (mon.0) 918 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.697920+0000 mon.a (mon.0) 919 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.700385+0000 mon.a (mon.0) 920 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.702785+0000 mon.a (mon.0) 921 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.704916+0000 mon.a (mon.0) 922 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.708067+0000 mon.a (mon.0) 923 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.711924+0000 mon.a (mon.0) 924 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:30 smithi129 bash[53756]: audit 2023-07-29T17:20:29.887838+0000 mgr.y (mgr.25169) 449 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:30.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.560290+0000 mon.a (mon.0) 905 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.569852+0000 mon.a (mon.0) 906 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.571621+0000 mon.a (mon.0) 907 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.572850+0000 mon.a (mon.0) 908 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: cephadm 2023-07-29T17:20:29.573960+0000 mgr.y (mgr.25169) 446 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: cluster 2023-07-29T17:20:29.575578+0000 mgr.y (mgr.25169) 447 : cluster 0 pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.581542+0000 mon.a (mon.0) 909 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.662940+0000 mon.a (mon.0) 910 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:20:30.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.664036+0000 mgr.y (mgr.25169) 448 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.673269+0000 mon.a (mon.0) 911 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.678210+0000 mon.a (mon.0) 912 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.680855+0000 mon.a (mon.0) 913 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.682880+0000 mon.a (mon.0) 914 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.686429+0000 mon.a (mon.0) 915 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.690411+0000 mon.a (mon.0) 916 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.693437+0000 mon.a (mon.0) 917 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.695721+0000 mon.a (mon.0) 918 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.697920+0000 mon.a (mon.0) 919 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.700385+0000 mon.a (mon.0) 920 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.702785+0000 mon.a (mon.0) 921 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.704916+0000 mon.a (mon.0) 922 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.708067+0000 mon.a (mon.0) 923 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.711924+0000 mon.a (mon.0) 924 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[60360]: audit 2023-07-29T17:20:29.887838+0000 mgr.y (mgr.25169) 449 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:30.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.560290+0000 mon.a (mon.0) 905 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.569852+0000 mon.a (mon.0) 906 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.571621+0000 mon.a (mon.0) 907 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.572850+0000 mon.a (mon.0) 908 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: cephadm 2023-07-29T17:20:29.573960+0000 mgr.y (mgr.25169) 446 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: cluster 2023-07-29T17:20:29.575578+0000 mgr.y (mgr.25169) 447 : cluster 0 pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.581542+0000 mon.a (mon.0) 909 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.662940+0000 mon.a (mon.0) 910 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.664036+0000 mgr.y (mgr.25169) 448 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-29T17:20:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.673269+0000 mon.a (mon.0) 911 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:30.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.678210+0000 mon.a (mon.0) 912 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.680855+0000 mon.a (mon.0) 913 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.682880+0000 mon.a (mon.0) 914 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.686429+0000 mon.a (mon.0) 915 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.690411+0000 mon.a (mon.0) 916 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.693437+0000 mon.a (mon.0) 917 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.695721+0000 mon.a (mon.0) 918 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.697920+0000 mon.a (mon.0) 919 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.700385+0000 mon.a (mon.0) 920 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.702785+0000 mon.a (mon.0) 921 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.704916+0000 mon.a (mon.0) 922 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.708067+0000 mon.a (mon.0) 923 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.711924+0000 mon.a (mon.0) 924 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:30 smithi078 bash[58614]: audit 2023-07-29T17:20:29.887838+0000 mgr.y (mgr.25169) 449 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:31 smithi129 bash[53756]: cluster 2023-07-29T17:20:30.568343+0000 mon.a (mon.0) 925 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-29T17:20:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:31 smithi129 bash[53756]: cluster 2023-07-29T17:20:30.568397+0000 mon.a (mon.0) 926 : cluster 1 Cluster is now healthy 2023-07-29T17:20:31.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:31 smithi078 bash[58614]: cluster 2023-07-29T17:20:30.568343+0000 mon.a (mon.0) 925 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-29T17:20:31.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:31 smithi078 bash[58614]: cluster 2023-07-29T17:20:30.568397+0000 mon.a (mon.0) 926 : cluster 1 Cluster is now healthy 2023-07-29T17:20:31.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:31 smithi078 bash[60360]: cluster 2023-07-29T17:20:30.568343+0000 mon.a (mon.0) 925 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-29T17:20:31.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:31 smithi078 bash[60360]: cluster 2023-07-29T17:20:30.568397+0000 mon.a (mon.0) 926 : cluster 1 Cluster is now healthy 2023-07-29T17:20:32.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:32 smithi129 bash[53756]: cluster 2023-07-29T17:20:31.576268+0000 mgr.y (mgr.25169) 450 : cluster 0 pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:32.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:32 smithi129 bash[53756]: cephadm 2023-07-29T17:20:32.235721+0000 mgr.y (mgr.25169) 451 : cephadm 1 Upgrade: Updating alertmanager.a 2023-07-29T17:20:32.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:32 smithi078 bash[58614]: cluster 2023-07-29T17:20:31.576268+0000 mgr.y (mgr.25169) 450 : cluster 0 pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:32.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:32 smithi078 bash[58614]: cephadm 2023-07-29T17:20:32.235721+0000 mgr.y (mgr.25169) 451 : cephadm 1 Upgrade: Updating alertmanager.a 2023-07-29T17:20:32.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:32 smithi078 bash[60360]: cluster 2023-07-29T17:20:31.576268+0000 mgr.y (mgr.25169) 450 : cluster 0 pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:32.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:32 smithi078 bash[60360]: cephadm 2023-07-29T17:20:32.235721+0000 mgr.y (mgr.25169) 451 : cephadm 1 Upgrade: Updating alertmanager.a 2023-07-29T17:20:33.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:33 smithi078 bash[58614]: cephadm 2023-07-29T17:20:32.247135+0000 mgr.y (mgr.25169) 452 : cephadm 1 Deploying daemon alertmanager.a on smithi078 2023-07-29T17:20:33.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:33 smithi078 bash[60360]: cephadm 2023-07-29T17:20:32.247135+0000 mgr.y (mgr.25169) 452 : cephadm 1 Deploying daemon alertmanager.a on smithi078 2023-07-29T17:20:33.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:33 smithi129 bash[53756]: cephadm 2023-07-29T17:20:32.247135+0000 mgr.y (mgr.25169) 452 : cephadm 1 Deploying daemon alertmanager.a on smithi078 2023-07-29T17:20:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:34 smithi078 bash[60360]: cluster 2023-07-29T17:20:33.577263+0000 mgr.y (mgr.25169) 453 : cluster 0 pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:34 smithi078 bash[58614]: cluster 2023-07-29T17:20:33.577263+0000 mgr.y (mgr.25169) 453 : cluster 0 pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:35.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:34 smithi129 bash[53756]: cluster 2023-07-29T17:20:33.577263+0000 mgr.y (mgr.25169) 453 : cluster 0 pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:36.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:36 smithi078 bash[58614]: cluster 2023-07-29T17:20:35.578302+0000 mgr.y (mgr.25169) 454 : cluster 0 pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:36.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:36 smithi078 bash[60360]: cluster 2023-07-29T17:20:35.578302+0000 mgr.y (mgr.25169) 454 : cluster 0 pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:36.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:36 smithi078 systemd[1]: Stopping Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:20:37.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:36 smithi129 bash[53756]: cluster 2023-07-29T17:20:35.578302+0000 mgr.y (mgr.25169) 454 : cluster 0 pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:37.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:37 smithi078 bash[55117]: level=info ts=2023-07-29T17:20:37.027Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:20:38.195 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:37 smithi078 bash[78821]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-alertmanager.a 2023-07-29T17:20:38.195 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:37 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-29T17:20:38.195 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:37 smithi078 systemd[1]: Stopped Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:20:38.695 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:38 smithi078 systemd[1]: Started Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:20:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:38 smithi129 bash[53756]: cluster 2023-07-29T17:20:37.579453+0000 mgr.y (mgr.25169) 455 : cluster 0 pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:38 smithi129 bash[53756]: audit 2023-07-29T17:20:38.270493+0000 mon.a (mon.0) 927 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:39.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:38 smithi129 bash[53756]: audit 2023-07-29T17:20:38.278225+0000 mon.a (mon.0) 928 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:39.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:38 smithi078 bash[58614]: cluster 2023-07-29T17:20:37.579453+0000 mgr.y (mgr.25169) 455 : cluster 0 pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:39.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:38 smithi078 bash[58614]: audit 2023-07-29T17:20:38.270493+0000 mon.a (mon.0) 927 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:39.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:38 smithi078 bash[58614]: audit 2023-07-29T17:20:38.278225+0000 mon.a (mon.0) 928 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:38 smithi078 bash[60360]: cluster 2023-07-29T17:20:37.579453+0000 mgr.y (mgr.25169) 455 : cluster 0 pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2023-07-29T17:20:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:38 smithi078 bash[60360]: audit 2023-07-29T17:20:38.270493+0000 mon.a (mon.0) 927 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:38 smithi078 bash[60360]: audit 2023-07-29T17:20:38.278225+0000 mon.a (mon.0) 928 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:39.945 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.635Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.635Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.637Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.78 port=9094 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.638Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.694Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.695Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.701Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2023-07-29T17:20:39.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:39 smithi078 bash[78981]: ts=2023-07-29T17:20:39.701Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2023-07-29T17:20:41.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:40 smithi129 bash[53756]: cluster 2023-07-29T17:20:39.580365+0000 mgr.y (mgr.25169) 456 : cluster 0 pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:41.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:40 smithi129 bash[53756]: audit 2023-07-29T17:20:39.894090+0000 mgr.y (mgr.25169) 457 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:40 smithi078 bash[58614]: cluster 2023-07-29T17:20:39.580365+0000 mgr.y (mgr.25169) 456 : cluster 0 pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:40 smithi078 bash[58614]: audit 2023-07-29T17:20:39.894090+0000 mgr.y (mgr.25169) 457 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:40 smithi078 bash[60360]: cluster 2023-07-29T17:20:39.580365+0000 mgr.y (mgr.25169) 456 : cluster 0 pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-29T17:20:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:40 smithi078 bash[60360]: audit 2023-07-29T17:20:39.894090+0000 mgr.y (mgr.25169) 457 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:41.682 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:20:41.945 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:41 smithi078 bash[78981]: ts=2023-07-29T17:20:41.639Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000646464s 2023-07-29T17:20:42.418 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:20:42.418 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 starting - - - - 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (9m) 23s ago 16m 26.5M - 6.7.4 557c83e11646 d9049963f3ca 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (84s) 23s ago 14m 46.6M - 3.6 b662b4d80c86 a64a43bc0b16 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (8m) 23s ago 21m 430M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (9m) 51s ago 22m 480M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (8m) 51s ago 23m 82.1M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (7m) 23s ago 21m 62.4M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:20:42.419 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7m) 51s ago 21m 61.0M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (59s) 51s ago 17m 3840k - 1.5.0 0da6a335fe13 4196bcd9b292 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (54s) 23s ago 17m 4320k - 1.5.0 0da6a335fe13 f9ecb2f5f5ff 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5m) 51s ago 20m 62.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5m) 51s ago 20m 65.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4m) 51s ago 19m 54.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4m) 51s ago 19m 61.4M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (3m) 23s ago 19m 61.9M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:20:42.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (3m) 23s ago 18m 55.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (3m) 23s ago 18m 54.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (2m) 23s ago 17m 56.1M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 running (31s) 23s ago 17m 28.1M - 2.43.0 a07b618ecd1d ce3cf859e4c4 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (2m) 51s ago 14m 84.0M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (113s) 23s ago 14m 84.2M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (2m) 51s ago 14m 84.2M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:20:42.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (106s) 23s ago 14m 84.2M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:20:42.860 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:20:42.860 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:20:42.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:20:42.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:42.860 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:20:42.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:20:42.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:20:42.862 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:20:42.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:20:42.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:20:42.862 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:20:43.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:42 smithi129 bash[53756]: cluster 2023-07-29T17:20:41.581208+0000 mgr.y (mgr.25169) 458 : cluster 0 pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:43.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:42 smithi129 bash[53756]: audit 2023-07-29T17:20:41.643769+0000 mgr.y (mgr.25169) 459 : audit 0 from='client.34971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:43.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:42 smithi129 bash[53756]: audit 2023-07-29T17:20:42.018192+0000 mgr.y (mgr.25169) 460 : audit 0 from='client.34977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:43.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:42 smithi078 bash[60360]: cluster 2023-07-29T17:20:41.581208+0000 mgr.y (mgr.25169) 458 : cluster 0 pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:43.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:42 smithi078 bash[60360]: audit 2023-07-29T17:20:41.643769+0000 mgr.y (mgr.25169) 459 : audit 0 from='client.34971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:43.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:42 smithi078 bash[60360]: audit 2023-07-29T17:20:42.018192+0000 mgr.y (mgr.25169) 460 : audit 0 from='client.34977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:43.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:42 smithi078 bash[58614]: cluster 2023-07-29T17:20:41.581208+0000 mgr.y (mgr.25169) 458 : cluster 0 pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:43.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:42 smithi078 bash[58614]: audit 2023-07-29T17:20:41.643769+0000 mgr.y (mgr.25169) 459 : audit 0 from='client.34971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:43.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:42 smithi078 bash[58614]: audit 2023-07-29T17:20:42.018192+0000 mgr.y (mgr.25169) 460 : audit 0 from='client.34977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:43.227 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:20:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:20:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:20:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:20:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:20:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "osd", 2023-07-29T17:20:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "iscsi", 2023-07-29T17:20:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "18/23 daemons upgraded", 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading alertmanager daemons", 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:20:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:20:43.666 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:20:44.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:43 smithi129 bash[53756]: audit 2023-07-29T17:20:42.378541+0000 mgr.y (mgr.25169) 461 : audit 0 from='client.44825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:44.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:43 smithi129 bash[53756]: audit 2023-07-29T17:20:42.834797+0000 mon.c (mon.2) 17 : audit 0 from='client.34989 172.21.15.78:0/2974087159' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:44.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:43 smithi129 bash[53756]: audit 2023-07-29T17:20:43.202550+0000 mgr.y (mgr.25169) 462 : audit 0 from='client.44837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:44.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:43 smithi129 bash[53756]: audit 2023-07-29T17:20:43.642451+0000 mon.a (mon.0) 929 : audit 0 from='client.35001 172.21.15.78:0/4116801561' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:20:44.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[58614]: audit 2023-07-29T17:20:42.378541+0000 mgr.y (mgr.25169) 461 : audit 0 from='client.44825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:44.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[58614]: audit 2023-07-29T17:20:42.834797+0000 mon.c (mon.2) 17 : audit 0 from='client.34989 172.21.15.78:0/2974087159' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:44.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[58614]: audit 2023-07-29T17:20:43.202550+0000 mgr.y (mgr.25169) 462 : audit 0 from='client.44837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:44.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[58614]: audit 2023-07-29T17:20:43.642451+0000 mon.a (mon.0) 929 : audit 0 from='client.35001 172.21.15.78:0/4116801561' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:20:44.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[60360]: audit 2023-07-29T17:20:42.378541+0000 mgr.y (mgr.25169) 461 : audit 0 from='client.44825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:44.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[60360]: audit 2023-07-29T17:20:42.834797+0000 mon.c (mon.2) 17 : audit 0 from='client.34989 172.21.15.78:0/2974087159' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:44.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[60360]: audit 2023-07-29T17:20:43.202550+0000 mgr.y (mgr.25169) 462 : audit 0 from='client.44837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:20:44.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:43 smithi078 bash[60360]: audit 2023-07-29T17:20:43.642451+0000 mon.a (mon.0) 929 : audit 0 from='client.35001 172.21.15.78:0/4116801561' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:20:45.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:44 smithi129 bash[53756]: cluster 2023-07-29T17:20:43.582666+0000 mgr.y (mgr.25169) 463 : cluster 0 pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:45.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:44 smithi129 bash[53756]: audit 2023-07-29T17:20:44.776918+0000 mon.a (mon.0) 930 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:45.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:44 smithi129 bash[53756]: audit 2023-07-29T17:20:44.785774+0000 mon.a (mon.0) 931 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:45.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:44 smithi078 bash[60360]: cluster 2023-07-29T17:20:43.582666+0000 mgr.y (mgr.25169) 463 : cluster 0 pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:45.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:44 smithi078 bash[60360]: audit 2023-07-29T17:20:44.776918+0000 mon.a (mon.0) 930 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:45.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:44 smithi078 bash[60360]: audit 2023-07-29T17:20:44.785774+0000 mon.a (mon.0) 931 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:45.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:44 smithi078 bash[58614]: cluster 2023-07-29T17:20:43.582666+0000 mgr.y (mgr.25169) 463 : cluster 0 pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:45.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:44 smithi078 bash[58614]: audit 2023-07-29T17:20:44.776918+0000 mon.a (mon.0) 930 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:45.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:44 smithi078 bash[58614]: audit 2023-07-29T17:20:44.785774+0000 mon.a (mon.0) 931 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:46 smithi129 bash[53756]: audit 2023-07-29T17:20:45.284555+0000 mon.a (mon.0) 932 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:46 smithi129 bash[53756]: audit 2023-07-29T17:20:45.295633+0000 mon.a (mon.0) 933 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:46 smithi129 bash[53756]: cluster 2023-07-29T17:20:45.583770+0000 mgr.y (mgr.25169) 464 : cluster 0 pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:46.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:46 smithi129 bash[53756]: audit 2023-07-29T17:20:46.095948+0000 mon.a (mon.0) 934 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:46 smithi129 bash[53756]: audit 2023-07-29T17:20:46.106974+0000 mon.a (mon.0) 935 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[58614]: audit 2023-07-29T17:20:45.284555+0000 mon.a (mon.0) 932 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[58614]: audit 2023-07-29T17:20:45.295633+0000 mon.a (mon.0) 933 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[58614]: cluster 2023-07-29T17:20:45.583770+0000 mgr.y (mgr.25169) 464 : cluster 0 pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:46.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[58614]: audit 2023-07-29T17:20:46.095948+0000 mon.a (mon.0) 934 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[58614]: audit 2023-07-29T17:20:46.106974+0000 mon.a (mon.0) 935 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[60360]: audit 2023-07-29T17:20:45.284555+0000 mon.a (mon.0) 932 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[60360]: audit 2023-07-29T17:20:45.295633+0000 mon.a (mon.0) 933 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[60360]: cluster 2023-07-29T17:20:45.583770+0000 mgr.y (mgr.25169) 464 : cluster 0 pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:46.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[60360]: audit 2023-07-29T17:20:46.095948+0000 mon.a (mon.0) 934 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:46.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:46 smithi078 bash[60360]: audit 2023-07-29T17:20:46.106974+0000 mon.a (mon.0) 935 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:48.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:48 smithi129 bash[53756]: cluster 2023-07-29T17:20:47.584900+0000 mgr.y (mgr.25169) 465 : cluster 0 pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:48.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:48 smithi078 bash[58614]: cluster 2023-07-29T17:20:47.584900+0000 mgr.y (mgr.25169) 465 : cluster 0 pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:48.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:48 smithi078 bash[60360]: cluster 2023-07-29T17:20:47.584900+0000 mgr.y (mgr.25169) 465 : cluster 0 pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:49.945 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:20:49 smithi078 bash[78981]: ts=2023-07-29T17:20:49.642Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003328814s 2023-07-29T17:20:51.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:50 smithi129 bash[53756]: cluster 2023-07-29T17:20:49.586214+0000 mgr.y (mgr.25169) 466 : cluster 0 pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:51.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:50 smithi129 bash[53756]: audit 2023-07-29T17:20:49.899774+0000 mgr.y (mgr.25169) 467 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:50 smithi078 bash[60360]: cluster 2023-07-29T17:20:49.586214+0000 mgr.y (mgr.25169) 466 : cluster 0 pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:51.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:50 smithi078 bash[60360]: audit 2023-07-29T17:20:49.899774+0000 mgr.y (mgr.25169) 467 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:51.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:50 smithi078 bash[58614]: cluster 2023-07-29T17:20:49.586214+0000 mgr.y (mgr.25169) 466 : cluster 0 pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:51.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:50 smithi078 bash[58614]: audit 2023-07-29T17:20:49.899774+0000 mgr.y (mgr.25169) 467 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:20:53.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:52 smithi129 bash[53756]: cluster 2023-07-29T17:20:51.587120+0000 mgr.y (mgr.25169) 468 : cluster 0 pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:53.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:52 smithi078 bash[60360]: cluster 2023-07-29T17:20:51.587120+0000 mgr.y (mgr.25169) 468 : cluster 0 pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:53.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:52 smithi078 bash[58614]: cluster 2023-07-29T17:20:51.587120+0000 mgr.y (mgr.25169) 468 : cluster 0 pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:55.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:54 smithi129 bash[53756]: cluster 2023-07-29T17:20:53.588664+0000 mgr.y (mgr.25169) 469 : cluster 0 pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:55.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:54 smithi078 bash[58614]: cluster 2023-07-29T17:20:53.588664+0000 mgr.y (mgr.25169) 469 : cluster 0 pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:55.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:54 smithi078 bash[60360]: cluster 2023-07-29T17:20:53.588664+0000 mgr.y (mgr.25169) 469 : cluster 0 pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:20:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: cluster 2023-07-29T17:20:55.589713+0000 mgr.y (mgr.25169) 470 : cluster 0 pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.058177+0000 mon.a (mon.0) 936 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.070567+0000 mon.a (mon.0) 937 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.072764+0000 mon.a (mon.0) 938 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.074602+0000 mon.a (mon.0) 939 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: cephadm 2023-07-29T17:20:56.075981+0000 mgr.y (mgr.25169) 471 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.085710+0000 mon.a (mon.0) 940 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.170567+0000 mon.a (mon.0) 941 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.171421+0000 mgr.y (mgr.25169) 472 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.179530+0000 mon.a (mon.0) 942 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.184402+0000 mon.a (mon.0) 943 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.187233+0000 mon.a (mon.0) 944 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.189304+0000 mon.a (mon.0) 945 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.192651+0000 mon.a (mon.0) 946 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.196606+0000 mon.a (mon.0) 947 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.199551+0000 mon.a (mon.0) 948 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.201701+0000 mon.a (mon.0) 949 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.203856+0000 mon.a (mon.0) 950 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.205944+0000 mon.a (mon.0) 951 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.208148+0000 mon.a (mon.0) 952 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.210243+0000 mon.a (mon.0) 953 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.213045+0000 mon.a (mon.0) 954 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.215439+0000 mon.a (mon.0) 955 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:57 smithi129 bash[53756]: audit 2023-07-29T17:20:56.218074+0000 mon.a (mon.0) 956 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: cluster 2023-07-29T17:20:55.589713+0000 mgr.y (mgr.25169) 470 : cluster 0 pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.058177+0000 mon.a (mon.0) 936 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.070567+0000 mon.a (mon.0) 937 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.072764+0000 mon.a (mon.0) 938 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.074602+0000 mon.a (mon.0) 939 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: cephadm 2023-07-29T17:20:56.075981+0000 mgr.y (mgr.25169) 471 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.085710+0000 mon.a (mon.0) 940 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.170567+0000 mon.a (mon.0) 941 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.171421+0000 mgr.y (mgr.25169) 472 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.179530+0000 mon.a (mon.0) 942 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.184402+0000 mon.a (mon.0) 943 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.187233+0000 mon.a (mon.0) 944 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.189304+0000 mon.a (mon.0) 945 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.192651+0000 mon.a (mon.0) 946 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.196606+0000 mon.a (mon.0) 947 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.199551+0000 mon.a (mon.0) 948 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.201701+0000 mon.a (mon.0) 949 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.203856+0000 mon.a (mon.0) 950 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.205944+0000 mon.a (mon.0) 951 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: cluster 2023-07-29T17:20:55.589713+0000 mgr.y (mgr.25169) 470 : cluster 0 pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:57.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.058177+0000 mon.a (mon.0) 936 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.070567+0000 mon.a (mon.0) 937 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.072764+0000 mon.a (mon.0) 938 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:20:57.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.074602+0000 mon.a (mon.0) 939 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:20:57.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: cephadm 2023-07-29T17:20:56.075981+0000 mgr.y (mgr.25169) 471 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.085710+0000 mon.a (mon.0) 940 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.170567+0000 mon.a (mon.0) 941 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.171421+0000 mgr.y (mgr.25169) 472 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.179530+0000 mon.a (mon.0) 942 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.184402+0000 mon.a (mon.0) 943 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.187233+0000 mon.a (mon.0) 944 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.189304+0000 mon.a (mon.0) 945 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.192651+0000 mon.a (mon.0) 946 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.196606+0000 mon.a (mon.0) 947 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.199551+0000 mon.a (mon.0) 948 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.201701+0000 mon.a (mon.0) 949 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.203856+0000 mon.a (mon.0) 950 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.205944+0000 mon.a (mon.0) 951 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.208148+0000 mon.a (mon.0) 952 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.210243+0000 mon.a (mon.0) 953 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.213045+0000 mon.a (mon.0) 954 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.215439+0000 mon.a (mon.0) 955 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[58614]: audit 2023-07-29T17:20:56.218074+0000 mon.a (mon.0) 956 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.208148+0000 mon.a (mon.0) 952 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.210243+0000 mon.a (mon.0) 953 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.213045+0000 mon.a (mon.0) 954 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.215439+0000 mon.a (mon.0) 955 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:57.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:57 smithi078 bash[60360]: audit 2023-07-29T17:20:56.218074+0000 mon.a (mon.0) 956 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:20:59.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:20:59 smithi078 bash[58614]: cluster 2023-07-29T17:20:57.590841+0000 mgr.y (mgr.25169) 473 : cluster 0 pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:59.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:20:59 smithi078 bash[60360]: cluster 2023-07-29T17:20:57.590841+0000 mgr.y (mgr.25169) 473 : cluster 0 pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:20:59.534 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:20:59 smithi129 bash[53756]: cluster 2023-07-29T17:20:57.590841+0000 mgr.y (mgr.25169) 473 : cluster 0 pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:00.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:00 smithi129 bash[53756]: cephadm 2023-07-29T17:20:58.991835+0000 mgr.y (mgr.25169) 474 : cephadm 1 Upgrade: Updating grafana.a 2023-07-29T17:21:00.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:00 smithi129 bash[53756]: cephadm 2023-07-29T17:20:59.012212+0000 mgr.y (mgr.25169) 475 : cephadm 1 Deploying daemon grafana.a on smithi129 2023-07-29T17:21:00.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:00 smithi078 bash[58614]: cephadm 2023-07-29T17:20:58.991835+0000 mgr.y (mgr.25169) 474 : cephadm 1 Upgrade: Updating grafana.a 2023-07-29T17:21:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:00 smithi078 bash[58614]: cephadm 2023-07-29T17:20:59.012212+0000 mgr.y (mgr.25169) 475 : cephadm 1 Deploying daemon grafana.a on smithi129 2023-07-29T17:21:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:00 smithi078 bash[60360]: cephadm 2023-07-29T17:20:58.991835+0000 mgr.y (mgr.25169) 474 : cephadm 1 Upgrade: Updating grafana.a 2023-07-29T17:21:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:00 smithi078 bash[60360]: cephadm 2023-07-29T17:20:59.012212+0000 mgr.y (mgr.25169) 475 : cephadm 1 Deploying daemon grafana.a on smithi129 2023-07-29T17:21:01.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:01 smithi129 bash[53756]: cluster 2023-07-29T17:20:59.592252+0000 mgr.y (mgr.25169) 476 : cluster 0 pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:01.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:01 smithi129 bash[53756]: audit 2023-07-29T17:20:59.909188+0000 mgr.y (mgr.25169) 477 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:01.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:01 smithi078 bash[58614]: cluster 2023-07-29T17:20:59.592252+0000 mgr.y (mgr.25169) 476 : cluster 0 pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:01.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:01 smithi078 bash[58614]: audit 2023-07-29T17:20:59.909188+0000 mgr.y (mgr.25169) 477 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:01.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:01 smithi078 bash[60360]: cluster 2023-07-29T17:20:59.592252+0000 mgr.y (mgr.25169) 476 : cluster 0 pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:01.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:01 smithi078 bash[60360]: audit 2023-07-29T17:20:59.909188+0000 mgr.y (mgr.25169) 477 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:03.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[53756]: cluster 2023-07-29T17:21:01.593212+0000 mgr.y (mgr.25169) 478 : cluster 0 pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:03.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:03 smithi078 bash[58614]: cluster 2023-07-29T17:21:01.593212+0000 mgr.y (mgr.25169) 478 : cluster 0 pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:03.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:03 smithi078 bash[60360]: cluster 2023-07-29T17:21:01.593212+0000 mgr.y (mgr.25169) 478 : cluster 0 pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:03.676 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 systemd[1]: Stopping Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:21:04.176 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2023-07-29T17:21:04.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2023-07-29T17:21:04.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2023-07-29T17:21:04.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2023-07-29T17:21:04.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2023-07-29T17:21:04.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2023-07-29T17:21:04.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2023-07-29T17:21:04.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2023-07-29T17:21:04.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2023-07-29T17:21:04.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2023-07-29T17:21:04.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped Stream Manager" 2023-07-29T17:21:04.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:03 smithi129 bash[45976]: t=2023-07-29T17:21:03+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2023-07-29T17:21:04.901 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:04 smithi129 bash[73947]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-grafana.a 2023-07-29T17:21:04.901 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:04 smithi129 bash[74025]: Error response from daemon: No such container: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-grafana.a 2023-07-29T17:21:04.901 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:04 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@grafana.a.service: Succeeded. 2023-07-29T17:21:04.902 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:04 smithi129 systemd[1]: Stopped Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:21:05.176 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:04 smithi129 systemd[1]: Started Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:21:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:05 smithi129 bash[53756]: cluster 2023-07-29T17:21:03.594688+0000 mgr.y (mgr.25169) 479 : cluster 0 pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:05 smithi129 bash[53756]: audit 2023-07-29T17:21:04.919005+0000 mon.a (mon.0) 957 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:05.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:05 smithi129 bash[53756]: audit 2023-07-29T17:21:04.928586+0000 mon.a (mon.0) 958 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:05.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:05 smithi078 bash[58614]: cluster 2023-07-29T17:21:03.594688+0000 mgr.y (mgr.25169) 479 : cluster 0 pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:05.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:05 smithi078 bash[58614]: audit 2023-07-29T17:21:04.919005+0000 mon.a (mon.0) 957 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:05.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:05 smithi078 bash[58614]: audit 2023-07-29T17:21:04.928586+0000 mon.a (mon.0) 958 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:05.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:05 smithi078 bash[60360]: cluster 2023-07-29T17:21:03.594688+0000 mgr.y (mgr.25169) 479 : cluster 0 pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:05.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:05 smithi078 bash[60360]: audit 2023-07-29T17:21:04.919005+0000 mon.a (mon.0) 957 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:05.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:05 smithi078 bash[60360]: audit 2023-07-29T17:21:04.928586+0000 mon.a (mon.0) 958 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:06.787 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.494946538Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2023-07-29T17:21:06.787 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495218755Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2023-07-29T17:21:06.788 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495245869Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2023-07-29T17:21:06.788 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495261334Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2023-07-29T17:21:06.788 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495274272Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2023-07-29T17:21:06.788 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495286146Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.49529863Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495311577Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495323426Z level=info msg="Path Home" path=/usr/share/grafana 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495335236Z level=info msg="Path Data" path=/var/lib/grafana 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495347699Z level=info msg="Path Logs" path=/var/log/grafana 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495361031Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2023-07-29T17:21:06.789 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495371555Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2023-07-29T17:21:06.790 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=settings t=2023-07-29T17:21:06.495383539Z level=info msg="App mode production" 2023-07-29T17:21:06.790 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=sqlstore t=2023-07-29T17:21:06.495458968Z level=info msg="Connecting to DB" dbtype=sqlite3 2023-07-29T17:21:06.790 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=sqlstore t=2023-07-29T17:21:06.495488035Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2023-07-29T17:21:06.790 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.496576638Z level=info msg="Starting DB migrations" 2023-07-29T17:21:06.790 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.496689821Z level=info msg="Executing migration" id="create migration_log table" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.526539379Z level=info msg="Executing migration" id="create user table" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.54993459Z level=info msg="Executing migration" id="add unique index user.login" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.589312007Z level=info msg="Executing migration" id="add unique index user.email" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.619151366Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.644648605Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.666094408Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2023-07-29T17:21:06.791 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.708149381Z level=info msg="Executing migration" id="create user table v2" 2023-07-29T17:21:06.792 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.738988807Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2023-07-29T17:21:06.792 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.764883781Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2023-07-29T17:21:07.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.790271383Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2023-07-29T17:21:07.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.857006053Z level=info msg="Executing migration" id="Drop old table user_v1" 2023-07-29T17:21:07.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.899278933Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2023-07-29T17:21:07.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.924445523Z level=info msg="Executing migration" id="Update user table charset" 2023-07-29T17:21:07.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.949777615Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2023-07-29T17:21:07.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:06 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:06.975085824Z level=info msg="Executing migration" id="Add missing user data" 2023-07-29T17:21:07.075 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.000386987Z level=info msg="Executing migration" id="Add is_disabled column to user" 2023-07-29T17:21:07.075 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.050939858Z level=info msg="Executing migration" id="Add index user.login/user.email" 2023-07-29T17:21:07.349 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[53756]: cluster 2023-07-29T17:21:05.595757+0000 mgr.y (mgr.25169) 480 : cluster 0 pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:07.349 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.092969671Z level=info msg="Executing migration" id="Add is_service_account column to user" 2023-07-29T17:21:07.350 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.135019463Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2023-07-29T17:21:07.350 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.160376505Z level=info msg="Executing migration" id="create temp user table v1-7" 2023-07-29T17:21:07.350 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.198226055Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2023-07-29T17:21:07.350 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.245636997Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2023-07-29T17:21:07.350 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.287586271Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2023-07-29T17:21:07.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:07 smithi078 bash[58614]: cluster 2023-07-29T17:21:05.595757+0000 mgr.y (mgr.25169) 480 : cluster 0 pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:07.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:07 smithi078 bash[60360]: cluster 2023-07-29T17:21:05.595757+0000 mgr.y (mgr.25169) 480 : cluster 0 pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:07.610 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.327236516Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2023-07-29T17:21:07.610 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.352557335Z level=info msg="Executing migration" id="Update temp_user table charset" 2023-07-29T17:21:07.610 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.377753457Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2023-07-29T17:21:07.610 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.444800452Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2023-07-29T17:21:07.610 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.470047736Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2023-07-29T17:21:07.611 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.495328876Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2023-07-29T17:21:07.611 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.520585003Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2023-07-29T17:21:07.611 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.545903929Z level=info msg="Executing migration" id="create temp_user v2" 2023-07-29T17:21:07.611 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.587773929Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2023-07-29T17:21:07.875 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.613086811Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2023-07-29T17:21:07.875 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.638317455Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2023-07-29T17:21:07.875 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.663575672Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2023-07-29T17:21:07.875 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.705697994Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2023-07-29T17:21:07.875 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.747701627Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2023-07-29T17:21:07.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.772993681Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2023-07-29T17:21:07.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.794979073Z level=info msg="Executing migration" id="create star table" 2023-07-29T17:21:07.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.816956925Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2023-07-29T17:21:07.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.852939619Z level=info msg="Executing migration" id="create org table v1" 2023-07-29T17:21:08.126 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.878064379Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2023-07-29T17:21:08.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.903316332Z level=info msg="Executing migration" id="create org_user table v1" 2023-07-29T17:21:08.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.928557193Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2023-07-29T17:21:08.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:07 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:07.953882592Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2023-07-29T17:21:08.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.037540173Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2023-07-29T17:21:08.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.062892675Z level=info msg="Executing migration" id="Update org table charset" 2023-07-29T17:21:08.127 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.104873209Z level=info msg="Executing migration" id="Update org_user table charset" 2023-07-29T17:21:08.386 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.12642781Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2023-07-29T17:21:08.386 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.165052387Z level=info msg="Executing migration" id="create dashboard table" 2023-07-29T17:21:08.387 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.207023444Z level=info msg="Executing migration" id="add index dashboard.account_id" 2023-07-29T17:21:08.387 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.229948911Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2023-07-29T17:21:08.387 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.271805241Z level=info msg="Executing migration" id="create dashboard_tag table" 2023-07-29T17:21:08.387 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.297143016Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2023-07-29T17:21:08.387 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.322529191Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-29T17:21:08.638 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.36438582Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2023-07-29T17:21:08.639 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.398062246Z level=info msg="Executing migration" id="create dashboard v2" 2023-07-29T17:21:08.639 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.423297363Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2023-07-29T17:21:08.639 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.448565913Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2023-07-29T17:21:08.640 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.473787211Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2023-07-29T17:21:08.640 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.515801614Z level=info msg="Executing migration" id="drop table dashboard_v1" 2023-07-29T17:21:08.640 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.541142714Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2023-07-29T17:21:08.641 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.616444314Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2023-07-29T17:21:08.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.641664528Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2023-07-29T17:21:08.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.666994563Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2023-07-29T17:21:08.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.708997293Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2023-07-29T17:21:08.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.758241019Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2023-07-29T17:21:08.928 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.812306237Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2023-07-29T17:21:08.928 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.854228566Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2023-07-29T17:21:08.928 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.904731017Z level=info msg="Executing migration" id="Update dashboard table charset" 2023-07-29T17:21:09.236 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[53756]: cluster 2023-07-29T17:21:07.596948+0000 mgr.y (mgr.25169) 481 : cluster 0 pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:09.237 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.930275214Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2023-07-29T17:21:09.237 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.958714971Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2023-07-29T17:21:09.237 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:08 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:08.984026817Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2023-07-29T17:21:09.237 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.009322703Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2023-07-29T17:21:09.237 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.051224031Z level=info msg="Executing migration" id="Add column uid in dashboard" 2023-07-29T17:21:09.238 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.0765728Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2023-07-29T17:21:09.238 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.118645185Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2023-07-29T17:21:09.238 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.144086581Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2023-07-29T17:21:09.238 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.214547851Z level=info msg="Executing migration" id="Update dashboard title length" 2023-07-29T17:21:09.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:09 smithi078 bash[58614]: cluster 2023-07-29T17:21:07.596948+0000 mgr.y (mgr.25169) 481 : cluster 0 pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:09.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:09 smithi078 bash[60360]: cluster 2023-07-29T17:21:07.596948+0000 mgr.y (mgr.25169) 481 : cluster 0 pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:09.491 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.256539384Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-29T17:21:09.492 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.298520964Z level=info msg="Executing migration" id="create dashboard_provisioning" 2023-07-29T17:21:09.492 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.323807601Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-29T17:21:09.492 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.34922426Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2023-07-29T17:21:09.493 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.374468928Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-29T17:21:09.493 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.419210726Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-29T17:21:09.493 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.444432633Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2023-07-29T17:21:09.493 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.469719023Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2023-07-29T17:21:09.772 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.495036999Z level=info msg="Executing migration" id="Add check_sum column" 2023-07-29T17:21:09.773 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.520290619Z level=info msg="Executing migration" id="Add index for dashboard_title" 2023-07-29T17:21:09.773 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.54548887Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2023-07-29T17:21:09.773 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.587436417Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2023-07-29T17:21:09.773 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.612800259Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2023-07-29T17:21:09.773 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.638207407Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2023-07-29T17:21:09.774 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.683481398Z level=info msg="Executing migration" id="create data_source table" 2023-07-29T17:21:09.774 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.708846386Z level=info msg="Executing migration" id="add index data_source.account_id" 2023-07-29T17:21:09.774 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.750751491Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2023-07-29T17:21:10.028 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.834422107Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2023-07-29T17:21:10.029 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.859808925Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2023-07-29T17:21:10.029 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.885061012Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2023-07-29T17:21:10.029 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.910437369Z level=info msg="Executing migration" id="create data_source table v2" 2023-07-29T17:21:10.029 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.956114297Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2023-07-29T17:21:10.029 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:09 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:09.981446422Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2023-07-29T17:21:10.317 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.006703275Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2023-07-29T17:21:10.318 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.048657129Z level=info msg="Executing migration" id="Add column with_credentials" 2023-07-29T17:21:10.318 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.073994234Z level=info msg="Executing migration" id="Add secure json data column" 2023-07-29T17:21:10.318 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.102176339Z level=info msg="Executing migration" id="Update data_source table charset" 2023-07-29T17:21:10.318 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.144205482Z level=info msg="Executing migration" id="Update initial version to 1" 2023-07-29T17:21:10.318 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.169473815Z level=info msg="Executing migration" id="Add read_only data column" 2023-07-29T17:21:10.318 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.21149114Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2023-07-29T17:21:10.319 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.253532662Z level=info msg="Executing migration" id="Update json_data with nulls" 2023-07-29T17:21:10.573 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.295617663Z level=info msg="Executing migration" id="Add uid column" 2023-07-29T17:21:10.573 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.320789445Z level=info msg="Executing migration" id="Update uid value" 2023-07-29T17:21:10.573 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.349535952Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2023-07-29T17:21:10.574 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.424850061Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2023-07-29T17:21:10.574 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.450222565Z level=info msg="Executing migration" id="create api_key table" 2023-07-29T17:21:10.574 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.475396537Z level=info msg="Executing migration" id="add index api_key.account_id" 2023-07-29T17:21:10.574 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.500680793Z level=info msg="Executing migration" id="add index api_key.key" 2023-07-29T17:21:10.574 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.525970372Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2023-07-29T17:21:10.862 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.551229896Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2023-07-29T17:21:10.862 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.593174176Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2023-07-29T17:21:10.862 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.621232821Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2023-07-29T17:21:10.862 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.646441638Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2023-07-29T17:21:10.862 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.688415545Z level=info msg="Executing migration" id="create api_key table v2" 2023-07-29T17:21:10.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.738852676Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2023-07-29T17:21:10.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.764046778Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2023-07-29T17:21:10.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.79790969Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2023-07-29T17:21:10.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.839986594Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2023-07-29T17:21:11.121 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.865152732Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2023-07-29T17:21:11.121 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.902589215Z level=info msg="Executing migration" id="Update api_key table charset" 2023-07-29T17:21:11.121 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:10 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:10.927828737Z level=info msg="Executing migration" id="Add expires to api_key table" 2023-07-29T17:21:11.121 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.003352439Z level=info msg="Executing migration" id="Add service account foreign key" 2023-07-29T17:21:11.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.045458339Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2023-07-29T17:21:11.122 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.074003993Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2023-07-29T17:21:11.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[53756]: cluster 2023-07-29T17:21:09.598287+0000 mgr.y (mgr.25169) 482 : cluster 0 pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:11.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[53756]: audit 2023-07-29T17:21:09.917147+0000 mgr.y (mgr.25169) 483 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:11.394 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.099404693Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2023-07-29T17:21:11.394 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.141402019Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2023-07-29T17:21:11.395 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.166728879Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2023-07-29T17:21:11.395 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.208946272Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2023-07-29T17:21:11.395 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.234084093Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2023-07-29T17:21:11.395 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.270549244Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-29T17:21:11.396 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.295882457Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-29T17:21:11.396 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.321080244Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2023-07-29T17:21:11.396 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.346231498Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2023-07-29T17:21:11.396 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.371534289Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2023-07-29T17:21:11.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:11 smithi078 bash[58614]: cluster 2023-07-29T17:21:09.598287+0000 mgr.y (mgr.25169) 482 : cluster 0 pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:11.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:11 smithi078 bash[58614]: audit 2023-07-29T17:21:09.917147+0000 mgr.y (mgr.25169) 483 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:11.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:11 smithi078 bash[60360]: cluster 2023-07-29T17:21:09.598287+0000 mgr.y (mgr.25169) 482 : cluster 0 pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:11.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:11 smithi078 bash[60360]: audit 2023-07-29T17:21:09.917147+0000 mgr.y (mgr.25169) 483 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:11.665 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.396853937Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2023-07-29T17:21:11.665 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.430239719Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2023-07-29T17:21:11.665 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.472511918Z level=info msg="Executing migration" id="create quota table v1" 2023-07-29T17:21:11.665 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.497771269Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-29T17:21:11.665 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.542723462Z level=info msg="Executing migration" id="Update quota table charset" 2023-07-29T17:21:11.666 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.618122778Z level=info msg="Executing migration" id="create plugin_setting table" 2023-07-29T17:21:11.920 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.643433782Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-29T17:21:11.921 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.685594017Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2023-07-29T17:21:11.921 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.727605559Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2023-07-29T17:21:11.921 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.7528284Z level=info msg="Executing migration" id="create session table" 2023-07-29T17:21:11.921 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.778164037Z level=info msg="Executing migration" id="Drop old table playlist table" 2023-07-29T17:21:11.921 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.803365678Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2023-07-29T17:21:11.922 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.848186147Z level=info msg="Executing migration" id="create playlist table v2" 2023-07-29T17:21:11.922 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.898984883Z level=info msg="Executing migration" id="create playlist item table v2" 2023-07-29T17:21:12.176 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.924331868Z level=info msg="Executing migration" id="Update playlist table charset" 2023-07-29T17:21:12.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.949563751Z level=info msg="Executing migration" id="Update playlist_item table charset" 2023-07-29T17:21:12.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:11 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:11.991451492Z level=info msg="Executing migration" id="drop preferences table v2" 2023-07-29T17:21:12.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.016922322Z level=info msg="Executing migration" id="drop preferences table v3" 2023-07-29T17:21:12.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.058900415Z level=info msg="Executing migration" id="create preferences table v3" 2023-07-29T17:21:12.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.086891954Z level=info msg="Executing migration" id="Update preferences table charset" 2023-07-29T17:21:12.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.128778489Z level=info msg="Executing migration" id="Add column team_id in preferences" 2023-07-29T17:21:12.482 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.204289535Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2023-07-29T17:21:12.482 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.246376033Z level=info msg="Executing migration" id="Add column week_start in preferences" 2023-07-29T17:21:12.482 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.288414103Z level=info msg="Executing migration" id="Add column preferences.json_data" 2023-07-29T17:21:12.483 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.33040268Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2023-07-29T17:21:12.483 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.359320442Z level=info msg="Executing migration" id="Add preferences index org_id" 2023-07-29T17:21:12.483 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.384565153Z level=info msg="Executing migration" id="Add preferences index user_id" 2023-07-29T17:21:12.483 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.409958293Z level=info msg="Executing migration" id="create alert table v1" 2023-07-29T17:21:12.483 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.435131612Z level=info msg="Executing migration" id="add index alert org_id & id " 2023-07-29T17:21:12.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.460320362Z level=info msg="Executing migration" id="add index alert state" 2023-07-29T17:21:12.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.485661227Z level=info msg="Executing migration" id="add index alert dashboard_id" 2023-07-29T17:21:12.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.510918727Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2023-07-29T17:21:12.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.55504628Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-29T17:21:12.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.578182488Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2023-07-29T17:21:12.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.611722427Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2023-07-29T17:21:12.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.637157222Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2023-07-29T17:21:12.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.662421555Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2023-07-29T17:21:12.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.704470987Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2023-07-29T17:21:12.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.788036073Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2023-07-29T17:21:13.079 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.829926915Z level=info msg="Executing migration" id="create alert_notification table v1" 2023-07-29T17:21:13.079 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.872106327Z level=info msg="Executing migration" id="Add column is_default" 2023-07-29T17:21:13.079 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.897211717Z level=info msg="Executing migration" id="Add column frequency" 2023-07-29T17:21:13.079 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.922647166Z level=info msg="Executing migration" id="Add column send_reminder" 2023-07-29T17:21:13.079 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.964539467Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2023-07-29T17:21:13.080 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:12 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:12.98976974Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2023-07-29T17:21:13.080 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.01519752Z level=info msg="Executing migration" id="Update alert table charset" 2023-07-29T17:21:13.080 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.057046361Z level=info msg="Executing migration" id="Update alert_notification table charset" 2023-07-29T17:21:13.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[53756]: cluster 2023-07-29T17:21:11.599228+0000 mgr.y (mgr.25169) 484 : cluster 0 pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:13.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[53756]: audit 2023-07-29T17:21:12.540036+0000 mon.a (mon.0) 959 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:13.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[53756]: audit 2023-07-29T17:21:12.550225+0000 mon.a (mon.0) 960 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:13.391 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.099620076Z level=info msg="Executing migration" id="create notification_journal table v1" 2023-07-29T17:21:13.391 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.141597288Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-29T17:21:13.391 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.183714212Z level=info msg="Executing migration" id="drop alert_notification_journal" 2023-07-29T17:21:13.392 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.225662301Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2023-07-29T17:21:13.392 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.250888541Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-29T17:21:13.392 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.292932036Z level=info msg="Executing migration" id="Add for to alert table" 2023-07-29T17:21:13.392 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.368203631Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2023-07-29T17:21:13.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:13 smithi078 bash[60360]: cluster 2023-07-29T17:21:11.599228+0000 mgr.y (mgr.25169) 484 : cluster 0 pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:13.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:13 smithi078 bash[60360]: audit 2023-07-29T17:21:12.540036+0000 mon.a (mon.0) 959 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:13.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:13 smithi078 bash[60360]: audit 2023-07-29T17:21:12.550225+0000 mon.a (mon.0) 960 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:13.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:13 smithi078 bash[58614]: cluster 2023-07-29T17:21:11.599228+0000 mgr.y (mgr.25169) 484 : cluster 0 pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:13.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:13 smithi078 bash[58614]: audit 2023-07-29T17:21:12.540036+0000 mon.a (mon.0) 959 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:13.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:13 smithi078 bash[58614]: audit 2023-07-29T17:21:12.550225+0000 mon.a (mon.0) 960 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:13.662 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.393420391Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2023-07-29T17:21:13.662 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.427194344Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2023-07-29T17:21:13.662 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.452480303Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2023-07-29T17:21:13.662 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.497554264Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2023-07-29T17:21:13.663 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.522855076Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2023-07-29T17:21:13.663 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.548077566Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2023-07-29T17:21:13.663 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.573367905Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2023-07-29T17:21:13.663 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.598634201Z level=info msg="Executing migration" id="Drop old annotation table v4" 2023-07-29T17:21:13.926 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.640541248Z level=info msg="Executing migration" id="create annotation table v5" 2023-07-29T17:21:13.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.665944189Z level=info msg="Executing migration" id="add index annotation 0 v3" 2023-07-29T17:21:13.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.708145402Z level=info msg="Executing migration" id="add index annotation 1 v3" 2023-07-29T17:21:13.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.750183868Z level=info msg="Executing migration" id="add index annotation 2 v3" 2023-07-29T17:21:13.928 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.775488039Z level=info msg="Executing migration" id="add index annotation 3 v3" 2023-07-29T17:21:13.928 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.817506821Z level=info msg="Executing migration" id="add index annotation 4 v3" 2023-07-29T17:21:13.928 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.859588541Z level=info msg="Executing migration" id="Update annotation table charset" 2023-07-29T17:21:14.068 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:21:14.217 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.929532816Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2023-07-29T17:21:14.217 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:13 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:13.976819853Z level=info msg="Executing migration" id="Drop category_id index" 2023-07-29T17:21:14.218 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.002089022Z level=info msg="Executing migration" id="Add column tags to annotation table" 2023-07-29T17:21:14.218 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.044040878Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2023-07-29T17:21:14.218 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.069363286Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-29T17:21:14.218 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.094572353Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2023-07-29T17:21:14.218 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.119800916Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2023-07-29T17:21:14.218 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.153483997Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2023-07-29T17:21:14.219 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.195491916Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2023-07-29T17:21:14.469 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[53756]: audit 2023-07-29T17:21:13.348645+0000 mon.a (mon.0) 961 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:14.470 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[53756]: audit 2023-07-29T17:21:13.357738+0000 mon.a (mon.0) 962 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:14.470 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[53756]: cluster 2023-07-29T17:21:13.600776+0000 mgr.y (mgr.25169) 485 : cluster 0 pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:14.470 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[53756]: audit 2023-07-29T17:21:14.028862+0000 mgr.y (mgr.25169) 486 : audit 0 from='client.35007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:14.470 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.237475428Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2023-07-29T17:21:14.470 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.262743284Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2023-07-29T17:21:14.471 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.304771082Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2023-07-29T17:21:14.471 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.330028648Z level=info msg="Executing migration" id="Add created time to annotation table" 2023-07-29T17:21:14.471 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.380431684Z level=info msg="Executing migration" id="Add updated time to annotation table" 2023-07-29T17:21:14.471 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.405733213Z level=info msg="Executing migration" id="Add index for created in annotation table" 2023-07-29T17:21:14.471 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.447759445Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2023-07-29T17:21:14.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[58614]: audit 2023-07-29T17:21:13.348645+0000 mon.a (mon.0) 961 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:14.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[58614]: audit 2023-07-29T17:21:13.357738+0000 mon.a (mon.0) 962 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:14.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[58614]: cluster 2023-07-29T17:21:13.600776+0000 mgr.y (mgr.25169) 485 : cluster 0 pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:14.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[58614]: audit 2023-07-29T17:21:14.028862+0000 mgr.y (mgr.25169) 486 : audit 0 from='client.35007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[60360]: audit 2023-07-29T17:21:13.348645+0000 mon.a (mon.0) 961 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:14.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[60360]: audit 2023-07-29T17:21:13.357738+0000 mon.a (mon.0) 962 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:14.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[60360]: cluster 2023-07-29T17:21:13.600776+0000 mgr.y (mgr.25169) 485 : cluster 0 pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:14.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:14 smithi078 bash[60360]: audit 2023-07-29T17:21:14.028862+0000 mgr.y (mgr.25169) 486 : audit 0 from='client.35007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:14.750 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.523221825Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2023-07-29T17:21:14.750 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.56505596Z level=info msg="Executing migration" id="Add epoch_end column" 2023-07-29T17:21:14.750 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.590404507Z level=info msg="Executing migration" id="Add index for epoch_end" 2023-07-29T17:21:14.750 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.618791867Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2023-07-29T17:21:14.750 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.643871504Z level=info msg="Executing migration" id="Move region to single row" 2023-07-29T17:21:14.751 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.685830169Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2023-07-29T17:21:14.751 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.728037177Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-29T17:21:14.807 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:21:14.807 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (36s) 29s ago 17m 15.1M - 0.25.0 c8568f914cd2 575415f1371d 2023-07-29T17:21:14.807 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (9s) 2s ago 17m 25.6M - 9.4.7 2c41d148cca3 360b40446815 2023-07-29T17:21:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (116s) 2s ago 15m 47.1M - 3.6 b662b4d80c86 a64a43bc0b16 2023-07-29T17:21:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (9m) 2s ago 21m 431M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:21:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (10m) 29s ago 23m 485M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:21:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (8m) 29s ago 23m 84.6M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:21:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (7m) 2s ago 22m 67.4M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:21:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (8m) 29s ago 22m 63.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (91s) 29s ago 17m 8787k - 1.5.0 0da6a335fe13 4196bcd9b292 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (87s) 2s ago 17m 9944k - 1.5.0 0da6a335fe13 f9ecb2f5f5ff 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6m) 29s ago 21m 62.6M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6m) 29s ago 20m 65.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5m) 29s ago 20m 54.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5m) 29s ago 20m 62.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (4m) 2s ago 19m 62.7M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:21:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (4m) 2s ago 19m 56.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (3m) 2s ago 18m 54.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (3m) 2s ago 18m 56.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 running (63s) 2s ago 17m 38.0M - 2.43.0 a07b618ecd1d ce3cf859e4c4 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (2m) 29s ago 15m 84.3M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (2m) 2s ago 15m 84.6M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (2m) 29s ago 15m 84.5M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:21:14.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (2m) 2s ago 15m 84.6M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:21:15.019 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.753297017Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-29T17:21:15.019 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.795259053Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-29T17:21:15.020 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.820552025Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-29T17:21:15.020 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.862421184Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2023-07-29T17:21:15.020 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.904767864Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2023-07-29T17:21:15.020 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.946698657Z level=info msg="Executing migration" id="create test_data table" 2023-07-29T17:21:15.020 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.972058068Z level=info msg="Executing migration" id="create dashboard_version table v1" 2023-07-29T17:21:15.021 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:14 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:14.997412945Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2023-07-29T17:21:15.244 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:15.245 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:21:15.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:21:15.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:21:15.247 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:21:15.279 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.022601339Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-29T17:21:15.280 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.064526146Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2023-07-29T17:21:15.280 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.139783477Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2023-07-29T17:21:15.280 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.165136268Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2023-07-29T17:21:15.280 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.207155095Z level=info msg="Executing migration" id="create team table" 2023-07-29T17:21:15.280 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.232385714Z level=info msg="Executing migration" id="add index team.org_id" 2023-07-29T17:21:15.280 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.257592928Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2023-07-29T17:21:15.566 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[53756]: audit 2023-07-29T17:21:14.403792+0000 mgr.y (mgr.25169) 487 : audit 0 from='client.35013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:15.566 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[53756]: audit 2023-07-29T17:21:14.767193+0000 mgr.y (mgr.25169) 488 : audit 0 from='client.44861 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:15.566 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[53756]: audit 2023-07-29T17:21:15.219344+0000 mon.a (mon.0) 963 : audit 0 from='client.35025 172.21.15.78:0/3170497111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:15.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.299655701Z level=info msg="Executing migration" id="create team member table" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.324929954Z level=info msg="Executing migration" id="add index team_member.org_id" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.375455052Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.400865831Z level=info msg="Executing migration" id="add index team_member.team_id" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.426106627Z level=info msg="Executing migration" id="Add column email to team table" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.45136425Z level=info msg="Executing migration" id="Add column external to team_member table" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.47657002Z level=info msg="Executing migration" id="Add column permission to team_member table" 2023-07-29T17:21:15.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.501909021Z level=info msg="Executing migration" id="create dashboard acl table" 2023-07-29T17:21:15.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.543852221Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2023-07-29T17:21:15.614 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:21:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577", 2023-07-29T17:21:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2023-07-29T17:21:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-29T17:21:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2023-07-29T17:21:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "osd", 2023-07-29T17:21:15.615 INFO:teuthology.orchestra.run.smithi078.stdout: "iscsi", 2023-07-29T17:21:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2023-07-29T17:21:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2023-07-29T17:21:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "mon" 2023-07-29T17:21:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-07-29T17:21:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "18/23 daemons upgraded", 2023-07-29T17:21:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading grafana daemons", 2023-07-29T17:21:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:21:15.618 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:21:15.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:15 smithi078 bash[58614]: audit 2023-07-29T17:21:14.403792+0000 mgr.y (mgr.25169) 487 : audit 0 from='client.35013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:15.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:15 smithi078 bash[58614]: audit 2023-07-29T17:21:14.767193+0000 mgr.y (mgr.25169) 488 : audit 0 from='client.44861 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:15.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:15 smithi078 bash[58614]: audit 2023-07-29T17:21:15.219344+0000 mon.a (mon.0) 963 : audit 0 from='client.35025 172.21.15.78:0/3170497111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:15.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:15 smithi078 bash[60360]: audit 2023-07-29T17:21:14.403792+0000 mgr.y (mgr.25169) 487 : audit 0 from='client.35013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:15.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:15 smithi078 bash[60360]: audit 2023-07-29T17:21:14.767193+0000 mgr.y (mgr.25169) 488 : audit 0 from='client.44861 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:15.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:15 smithi078 bash[60360]: audit 2023-07-29T17:21:15.219344+0000 mon.a (mon.0) 963 : audit 0 from='client.35025 172.21.15.78:0/3170497111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:15.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.569195108Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-29T17:21:15.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.594458631Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-29T17:21:15.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.619666433Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2023-07-29T17:21:15.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.653380888Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2023-07-29T17:21:15.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.678549955Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2023-07-29T17:21:15.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.745749673Z level=info msg="Executing migration" id="add index dashboard_permission" 2023-07-29T17:21:15.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.773684413Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2023-07-29T17:21:15.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.798914989Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2023-07-29T17:21:16.053 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:21:16.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.840909737Z level=info msg="Executing migration" id="create tag table" 2023-07-29T17:21:16.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.866335139Z level=info msg="Executing migration" id="add index tag.key_value" 2023-07-29T17:21:16.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.891607192Z level=info msg="Executing migration" id="create login attempt table" 2023-07-29T17:21:16.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.916847055Z level=info msg="Executing migration" id="add index login_attempt.username" 2023-07-29T17:21:16.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.942361329Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2023-07-29T17:21:16.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:15 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:15.967627677Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-29T17:21:16.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.026294116Z level=info msg="Executing migration" id="create login_attempt v2" 2023-07-29T17:21:16.334 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.051640011Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2023-07-29T17:21:16.334 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.076805862Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2023-07-29T17:21:16.334 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.102040208Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2023-07-29T17:21:16.334 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.127453341Z level=info msg="Executing migration" id="create user auth table" 2023-07-29T17:21:16.335 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.169347847Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-29T17:21:16.335 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.211285626Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2023-07-29T17:21:16.335 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.236638524Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2023-07-29T17:21:16.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[53756]: audit 2023-07-29T17:21:15.589076+0000 mgr.y (mgr.25169) 489 : audit 0 from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:16.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[53756]: cluster 2023-07-29T17:21:15.601792+0000 mgr.y (mgr.25169) 490 : cluster 0 pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:16.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[53756]: audit 2023-07-29T17:21:16.028736+0000 mon.a (mon.0) 964 : audit 0 from='client.35037 172.21.15.78:0/2471513328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.311937852Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.362311973Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.40437631Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.429736954Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.45494535Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.48015358Z level=info msg="Executing migration" id="create server_lock table" 2023-07-29T17:21:16.604 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.505423902Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2023-07-29T17:21:16.605 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.530803345Z level=info msg="Executing migration" id="create user auth token table" 2023-07-29T17:21:16.605 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.5559026Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2023-07-29T17:21:16.605 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.581278434Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2023-07-29T17:21:16.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:16 smithi078 bash[58614]: audit 2023-07-29T17:21:15.589076+0000 mgr.y (mgr.25169) 489 : audit 0 from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:16.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:16 smithi078 bash[58614]: cluster 2023-07-29T17:21:15.601792+0000 mgr.y (mgr.25169) 490 : cluster 0 pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:16.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:16 smithi078 bash[58614]: audit 2023-07-29T17:21:16.028736+0000 mon.a (mon.0) 964 : audit 0 from='client.35037 172.21.15.78:0/2471513328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:21:16.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:16 smithi078 bash[60360]: audit 2023-07-29T17:21:15.589076+0000 mgr.y (mgr.25169) 489 : audit 0 from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:16.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:16 smithi078 bash[60360]: cluster 2023-07-29T17:21:15.601792+0000 mgr.y (mgr.25169) 490 : cluster 0 pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:16.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:16 smithi078 bash[60360]: audit 2023-07-29T17:21:16.028736+0000 mon.a (mon.0) 964 : audit 0 from='client.35037 172.21.15.78:0/2471513328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:21:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.623276735Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2023-07-29T17:21:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.648626689Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2023-07-29T17:21:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.67376339Z level=info msg="Executing migration" id="create cache_data table" 2023-07-29T17:21:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.699156844Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2023-07-29T17:21:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.741108982Z level=info msg="Executing migration" id="create short_url table v1" 2023-07-29T17:21:16.910 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.766416246Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2023-07-29T17:21:16.911 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.791542962Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2023-07-29T17:21:16.911 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.816859409Z level=info msg="Executing migration" id="delete alert_definition table" 2023-07-29T17:21:16.911 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.887964987Z level=info msg="Executing migration" id="recreate alert_definition table" 2023-07-29T17:21:17.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.913430581Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2023-07-29T17:21:17.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.955395183Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2023-07-29T17:21:17.162 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:16 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:16.980647206Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2023-07-29T17:21:17.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.005911841Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2023-07-29T17:21:17.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.047766268Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2023-07-29T17:21:17.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.073154611Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2023-07-29T17:21:17.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.098512935Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2023-07-29T17:21:17.163 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.140439521Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2023-07-29T17:21:17.426 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.165620251Z level=info msg="Executing migration" id="drop alert_definition table" 2023-07-29T17:21:17.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.207614469Z level=info msg="Executing migration" id="delete alert_definition_version table" 2023-07-29T17:21:17.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.249626956Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2023-07-29T17:21:17.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.291618511Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2023-07-29T17:21:17.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.333636324Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2023-07-29T17:21:17.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.358936182Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2023-07-29T17:21:17.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.384554877Z level=info msg="Executing migration" id="drop alert_definition_version table" 2023-07-29T17:21:17.742 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.459930412Z level=info msg="Executing migration" id="create alert_instance table" 2023-07-29T17:21:17.742 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.485179805Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.510463025Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.535664869Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.57764181Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.602919909Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.62819961Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.686812596Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2023-07-29T17:21:17.743 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.720469811Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2023-07-29T17:21:17.998 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.762332072Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2023-07-29T17:21:17.998 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.804443533Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2023-07-29T17:21:17.998 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.82981347Z level=info msg="Executing migration" id="create alert_rule table" 2023-07-29T17:21:17.998 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.855073956Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2023-07-29T17:21:17.999 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.880274488Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2023-07-29T17:21:17.999 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.908666131Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2023-07-29T17:21:17.999 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.933943657Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2023-07-29T17:21:17.999 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:17 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:17.975819084Z level=info msg="Executing migration" id="add column for to alert_rule" 2023-07-29T17:21:18.258 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.051257707Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2023-07-29T17:21:18.259 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.076762897Z level=info msg="Executing migration" id="add column labels to alert_rule" 2023-07-29T17:21:18.259 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.102070355Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2023-07-29T17:21:18.259 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.127404399Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2023-07-29T17:21:18.259 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.169265385Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2023-07-29T17:21:18.259 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.211305968Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2023-07-29T17:21:18.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.236501437Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2023-07-29T17:21:18.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.261901627Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2023-07-29T17:21:18.545 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.303823614Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2023-07-29T17:21:18.545 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.446923473Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2023-07-29T17:21:18.616 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.522368295Z level=info msg="Executing migration" id="create alert_rule_version table" 2023-07-29T17:21:18.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[53756]: cluster 2023-07-29T17:21:17.602883+0000 mgr.y (mgr.25169) 491 : cluster 0 pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:18.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.594695014Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2023-07-29T17:21:18.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:18 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:18.764164131Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2023-07-29T17:21:18.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:18 smithi078 bash[58614]: cluster 2023-07-29T17:21:17.602883+0000 mgr.y (mgr.25169) 491 : cluster 0 pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:18.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:18 smithi078 bash[60360]: cluster 2023-07-29T17:21:17.602883+0000 mgr.y (mgr.25169) 491 : cluster 0 pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:19.426 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:19 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:19.093518509Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2023-07-29T17:21:19.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:19 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:19.287266176Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2023-07-29T17:21:19.721 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:19 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:19.442217409Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2023-07-29T17:21:20.003 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:19 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:19.700057511Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2023-07-29T17:21:20.003 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:19 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:19.800978208Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2023-07-29T17:21:20.305 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:19 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:19.981268982Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2023-07-29T17:21:20.305 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.05689105Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2023-07-29T17:21:20.305 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.149659299Z level=info msg="Executing migration" id=create_alert_configuration_table 2023-07-29T17:21:20.305 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.224318813Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2023-07-29T17:21:20.305 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.283059982Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2023-07-29T17:21:20.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.341782835Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2023-07-29T17:21:20.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.459010295Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2023-07-29T17:21:20.843 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.534728477Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2023-07-29T17:21:20.843 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.559759768Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2023-07-29T17:21:20.843 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.602648914Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2023-07-29T17:21:20.843 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.669896045Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2023-07-29T17:21:20.844 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.748624068Z level=info msg="Executing migration" id="create provenance_type table" 2023-07-29T17:21:21.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:20 smithi078 bash[58614]: cluster 2023-07-29T17:21:19.604209+0000 mgr.y (mgr.25169) 492 : cluster 0 pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:21.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:20 smithi078 bash[58614]: audit 2023-07-29T17:21:19.928535+0000 mgr.y (mgr.25169) 493 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:21.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:20 smithi078 bash[60360]: cluster 2023-07-29T17:21:19.604209+0000 mgr.y (mgr.25169) 492 : cluster 0 pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:21.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:20 smithi078 bash[60360]: audit 2023-07-29T17:21:19.928535+0000 mgr.y (mgr.25169) 493 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:21.224 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[53756]: cluster 2023-07-29T17:21:19.604209+0000 mgr.y (mgr.25169) 492 : cluster 0 pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:21.224 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[53756]: audit 2023-07-29T17:21:19.928535+0000 mgr.y (mgr.25169) 493 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:21.224 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.821129001Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2023-07-29T17:21:21.224 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:20 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:20.94644359Z level=info msg="Executing migration" id="create alert_image table" 2023-07-29T17:21:21.507 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.206180371Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2023-07-29T17:21:21.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.485675174Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2023-07-29T17:21:21.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.627726599Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2023-07-29T17:21:21.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.728400386Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2023-07-29T17:21:22.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.795617066Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2023-07-29T17:21:22.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.795989678Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2023-07-29T17:21:22.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.83825421Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2023-07-29T17:21:22.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:21 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:21.880466062Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2023-07-29T17:21:22.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.039320158Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2023-07-29T17:21:22.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.039888017Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2023-07-29T17:21:22.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.040356161Z level=info msg="alerts found to migrate" alerts=0 2023-07-29T17:21:22.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.315793811Z level=info msg="Executing migration" id="create library_element table v1" 2023-07-29T17:21:22.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.375206045Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2023-07-29T17:21:22.678 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.4506044Z level=info msg="Executing migration" id="create library_element_connection table v1" 2023-07-29T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[53756]: cluster 2023-07-29T17:21:21.605178+0000 mgr.y (mgr.25169) 494 : cluster 0 pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:22.985 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.688648227Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2023-07-29T17:21:22.985 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.818626403Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2023-07-29T17:21:23.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:22 smithi078 bash[58614]: cluster 2023-07-29T17:21:21.605178+0000 mgr.y (mgr.25169) 494 : cluster 0 pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:23.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:22 smithi078 bash[60360]: cluster 2023-07-29T17:21:21.605178+0000 mgr.y (mgr.25169) 494 : cluster 0 pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:23.242 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:22 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:22.962471395Z level=info msg="Executing migration" id="increase max description length to 2048" 2023-07-29T17:21:23.242 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.077854497Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2023-07-29T17:21:23.242 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.163308184Z level=info msg="Executing migration" id="create data_keys table" 2023-07-29T17:21:23.509 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.220406141Z level=info msg="Executing migration" id="create secrets table" 2023-07-29T17:21:23.509 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.290322325Z level=info msg="Executing migration" id="rename data_keys name column to id" 2023-07-29T17:21:23.509 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.390296641Z level=info msg="Executing migration" id="add name column into data_keys" 2023-07-29T17:21:23.509 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.43899227Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2023-07-29T17:21:23.509 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.46532234Z level=info msg="Executing migration" id="rename data_keys name column to label" 2023-07-29T17:21:23.840 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.557382377Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2023-07-29T17:21:23.840 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.632834431Z level=info msg="Executing migration" id="create kv_store table v1" 2023-07-29T17:21:23.841 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.658129707Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2023-07-29T17:21:23.841 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.683437041Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2023-07-29T17:21:23.841 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.708728161Z level=info msg="Executing migration" id="create permission table" 2023-07-29T17:21:23.841 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.750697101Z level=info msg="Executing migration" id="add unique index permission.role_id" 2023-07-29T17:21:23.842 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.79289745Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2023-07-29T17:21:23.842 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.818176124Z level=info msg="Executing migration" id="create role table" 2023-07-29T17:21:24.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.86020394Z level=info msg="Executing migration" id="add column display_name" 2023-07-29T17:21:24.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.885403577Z level=info msg="Executing migration" id="add column group_name" 2023-07-29T17:21:24.160 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.910805518Z level=info msg="Executing migration" id="add index role.org_id" 2023-07-29T17:21:24.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:23 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:23.973810943Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2023-07-29T17:21:24.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.024020082Z level=info msg="Executing migration" id="add index role_org_id_uid" 2023-07-29T17:21:24.161 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.066257689Z level=info msg="Executing migration" id="create team role table" 2023-07-29T17:21:24.426 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.138472867Z level=info msg="Executing migration" id="add index team_role.org_id" 2023-07-29T17:21:24.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.175025778Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2023-07-29T17:21:24.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.217043621Z level=info msg="Executing migration" id="add index team_role.team_id" 2023-07-29T17:21:24.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.259275729Z level=info msg="Executing migration" id="create user role table" 2023-07-29T17:21:24.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.301145158Z level=info msg="Executing migration" id="add index user_role.org_id" 2023-07-29T17:21:24.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.378974326Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2023-07-29T17:21:24.863 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.487310398Z level=info msg="Executing migration" id="add index user_role.user_id" 2023-07-29T17:21:24.864 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.619483732Z level=info msg="Executing migration" id="create builtin role table" 2023-07-29T17:21:25.153 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[53756]: cluster 2023-07-29T17:21:23.606730+0000 mgr.y (mgr.25169) 495 : cluster 0 pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:25.154 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:24 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:24.853144724Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2023-07-29T17:21:25.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:24 smithi078 bash[58614]: cluster 2023-07-29T17:21:23.606730+0000 mgr.y (mgr.25169) 495 : cluster 0 pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:25.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:24 smithi078 bash[60360]: cluster 2023-07-29T17:21:23.606730+0000 mgr.y (mgr.25169) 495 : cluster 0 pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:25.426 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.131757337Z level=info msg="Executing migration" id="add index builtin_role.name" 2023-07-29T17:21:25.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.273402441Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2023-07-29T17:21:25.427 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.381117772Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2023-07-29T17:21:25.744 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.448416961Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2023-07-29T17:21:25.744 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.491499301Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2023-07-29T17:21:25.745 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.533343913Z level=info msg="Executing migration" id="add unique index role.uid" 2023-07-29T17:21:25.745 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.575485699Z level=info msg="Executing migration" id="create seed assignment table" 2023-07-29T17:21:25.745 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.629685374Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2023-07-29T17:21:25.745 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.671692866Z level=info msg="Executing migration" id="add column hidden to role table" 2023-07-29T17:21:25.745 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.696980429Z level=info msg="Executing migration" id="create query_history table v1" 2023-07-29T17:21:25.745 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.722278806Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2023-07-29T17:21:25.996 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.772843577Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2023-07-29T17:21:25.997 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.814739179Z level=info msg="Executing migration" id="teams permissions migration" 2023-07-29T17:21:25.997 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.856972303Z level=info msg="Executing migration" id="dashboard permissions" 2023-07-29T17:21:25.997 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.921756065Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2023-07-29T17:21:25.998 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.974493316Z level=info msg="Executing migration" id="drop managed folder create actions" 2023-07-29T17:21:26.274 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:25 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:25.999717231Z level=info msg="Executing migration" id="alerting notification permissions" 2023-07-29T17:21:26.274 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.025100573Z level=info msg="Executing migration" id="create query_history_star table v1" 2023-07-29T17:21:26.274 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.067064556Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2023-07-29T17:21:26.275 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.09244302Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2023-07-29T17:21:26.275 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.117658992Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2023-07-29T17:21:26.275 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.142982371Z level=info msg="Executing migration" id="create correlation table v1" 2023-07-29T17:21:26.275 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.185081341Z level=info msg="Executing migration" id="add index correlations.uid" 2023-07-29T17:21:26.275 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.210324039Z level=info msg="Executing migration" id="add index correlations.source_uid" 2023-07-29T17:21:26.275 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.252327788Z level=info msg="Executing migration" id="add correlation config column" 2023-07-29T17:21:26.543 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.294285624Z level=info msg="Executing migration" id="create entity_events table" 2023-07-29T17:21:26.543 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.319681775Z level=info msg="Executing migration" id="create dashboard public config v1" 2023-07-29T17:21:26.543 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.344979291Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2023-07-29T17:21:26.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.345316228Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2023-07-29T17:21:26.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.37019486Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-29T17:21:26.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.370586789Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-29T17:21:26.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.403705088Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2023-07-29T17:21:26.544 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.429119006Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2023-07-29T17:21:26.545 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.454427776Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2023-07-29T17:21:26.545 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.521357322Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-29T17:21:26.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.56341736Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2023-07-29T17:21:26.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.60557004Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2023-07-29T17:21:26.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.63118231Z level=info msg="Executing migration" id="Drop public config table" 2023-07-29T17:21:26.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.656460933Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2023-07-29T17:21:26.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.715083532Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2023-07-29T17:21:26.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.760917571Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2023-07-29T17:21:26.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.786223117Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2023-07-29T17:21:26.810 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.518431+0000 mon.a (mon.0) 965 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.811 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.530571+0000 mon.a (mon.0) 966 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.811 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.532672+0000 mon.a (mon.0) 967 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.811 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.534538+0000 mon.a (mon.0) 968 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: cephadm 2023-07-29T17:21:25.536045+0000 mgr.y (mgr.25169) 496 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.545628+0000 mon.a (mon.0) 969 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: cluster 2023-07-29T17:21:25.607810+0000 mgr.y (mgr.25169) 497 : cluster 0 pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:26.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.629047+0000 mon.a (mon.0) 970 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:21:26.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.630055+0000 mgr.y (mgr.25169) 498 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:21:26.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.632158+0000 mon.a (mon.0) 971 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.632666+0000 mgr.y (mgr.25169) 499 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.643854+0000 mon.a (mon.0) 972 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.673792+0000 mon.a (mon.0) 973 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.678725+0000 mon.a (mon.0) 974 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.681179+0000 mon.a (mon.0) 975 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.683461+0000 mon.a (mon.0) 976 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.686858+0000 mon.a (mon.0) 977 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.690051+0000 mon.a (mon.0) 978 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.692915+0000 mon.a (mon.0) 979 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.694883+0000 mon.a (mon.0) 980 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.696787+0000 mon.a (mon.0) 981 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.698708+0000 mon.a (mon.0) 982 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.700791+0000 mon.a (mon.0) 983 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.702887+0000 mon.a (mon.0) 984 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.705727+0000 mon.a (mon.0) 985 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.708228+0000 mon.a (mon.0) 986 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.710814+0000 mon.a (mon.0) 987 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.713438+0000 mon.a (mon.0) 988 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.715618+0000 mon.a (mon.0) 989 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.717820+0000 mon.a (mon.0) 990 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: cephadm 2023-07-29T17:21:25.719062+0000 mgr.y (mgr.25169) 500 : cephadm 1 Upgrade: Finalizing container_image settings 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.729146+0000 mon.a (mon.0) 991 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.730930+0000 mon.a (mon.0) 992 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.738527+0000 mon.a (mon.0) 993 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.740405+0000 mon.a (mon.0) 994 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.749212+0000 mon.a (mon.0) 995 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.750981+0000 mon.a (mon.0) 996 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2023-07-29T17:21:26.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.758542+0000 mon.a (mon.0) 997 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.760075+0000 mon.a (mon.0) 998 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.767967+0000 mon.a (mon.0) 999 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.769496+0000 mon.a (mon.0) 1000 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.778171+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.780057+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.787701+0000 mon.a (mon.0) 1003 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2023-07-29T17:21:26.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.789275+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.798888+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.800344+0000 mon.a (mon.0) 1006 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.801701+0000 mon.a (mon.0) 1007 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.811490+0000 mon.a (mon.0) 1008 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.813076+0000 mon.a (mon.0) 1009 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.821903+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2023-07-29T17:21:26.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.823332+0000 mon.a (mon.0) 1011 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2023-07-29T17:21:26.819 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.829642+0000 mon.a (mon.0) 1012 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2023-07-29T17:21:26.819 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.830809+0000 mon.a (mon.0) 1013 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.819 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.831817+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.819 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.833107+0000 mon.a (mon.0) 1015 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.834178+0000 mon.a (mon.0) 1016 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.835250+0000 mon.a (mon.0) 1017 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.836318+0000 mon.a (mon.0) 1018 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.837485+0000 mon.a (mon.0) 1019 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.842996+0000 mon.a (mon.0) 1020 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: cephadm 2023-07-29T17:21:25.844266+0000 mgr.y (mgr.25169) 501 : cephadm 1 Upgrade: Complete! 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.845144+0000 mon.a (mon.0) 1021 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.849437+0000 mon.a (mon.0) 1022 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.851327+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.852650+0000 mon.a (mon.0) 1024 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: cephadm 2023-07-29T17:21:25.853860+0000 mgr.y (mgr.25169) 502 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.860321+0000 mon.a (mon.0) 1025 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.946576+0000 mon.a (mon.0) 1026 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.948263+0000 mon.a (mon.0) 1027 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: cephadm 2023-07-29T17:21:25.949456+0000 mgr.y (mgr.25169) 503 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[53756]: audit 2023-07-29T17:21:25.957822+0000 mon.a (mon.0) 1028 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.518431+0000 mon.a (mon.0) 965 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.530571+0000 mon.a (mon.0) 966 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.532672+0000 mon.a (mon.0) 967 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.534538+0000 mon.a (mon.0) 968 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: cephadm 2023-07-29T17:21:25.536045+0000 mgr.y (mgr.25169) 496 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.545628+0000 mon.a (mon.0) 969 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: cluster 2023-07-29T17:21:25.607810+0000 mgr.y (mgr.25169) 497 : cluster 0 pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:26.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.629047+0000 mon.a (mon.0) 970 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:21:26.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.630055+0000 mgr.y (mgr.25169) 498 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:21:26.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.632158+0000 mon.a (mon.0) 971 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:21:26.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.632666+0000 mgr.y (mgr.25169) 499 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.643854+0000 mon.a (mon.0) 972 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.673792+0000 mon.a (mon.0) 973 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.678725+0000 mon.a (mon.0) 974 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.681179+0000 mon.a (mon.0) 975 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.683461+0000 mon.a (mon.0) 976 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.686858+0000 mon.a (mon.0) 977 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.690051+0000 mon.a (mon.0) 978 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.692915+0000 mon.a (mon.0) 979 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.518431+0000 mon.a (mon.0) 965 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.530571+0000 mon.a (mon.0) 966 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.532672+0000 mon.a (mon.0) 967 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.534538+0000 mon.a (mon.0) 968 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: cephadm 2023-07-29T17:21:25.536045+0000 mgr.y (mgr.25169) 496 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.545628+0000 mon.a (mon.0) 969 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: cluster 2023-07-29T17:21:25.607810+0000 mgr.y (mgr.25169) 497 : cluster 0 pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.629047+0000 mon.a (mon.0) 970 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.630055+0000 mgr.y (mgr.25169) 498 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.632158+0000 mon.a (mon.0) 971 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:21:26.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.632666+0000 mgr.y (mgr.25169) 499 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129:3000"}]: dispatch 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.643854+0000 mon.a (mon.0) 972 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.673792+0000 mon.a (mon.0) 973 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.678725+0000 mon.a (mon.0) 974 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.681179+0000 mon.a (mon.0) 975 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.683461+0000 mon.a (mon.0) 976 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.686858+0000 mon.a (mon.0) 977 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.690051+0000 mon.a (mon.0) 978 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.692915+0000 mon.a (mon.0) 979 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.694883+0000 mon.a (mon.0) 980 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.696787+0000 mon.a (mon.0) 981 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.698708+0000 mon.a (mon.0) 982 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.700791+0000 mon.a (mon.0) 983 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.702887+0000 mon.a (mon.0) 984 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.705727+0000 mon.a (mon.0) 985 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.708228+0000 mon.a (mon.0) 986 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.710814+0000 mon.a (mon.0) 987 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.713438+0000 mon.a (mon.0) 988 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.715618+0000 mon.a (mon.0) 989 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.717820+0000 mon.a (mon.0) 990 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: cephadm 2023-07-29T17:21:25.719062+0000 mgr.y (mgr.25169) 500 : cephadm 1 Upgrade: Finalizing container_image settings 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.729146+0000 mon.a (mon.0) 991 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.730930+0000 mon.a (mon.0) 992 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2023-07-29T17:21:26.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.738527+0000 mon.a (mon.0) 993 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2023-07-29T17:21:26.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.740405+0000 mon.a (mon.0) 994 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.749212+0000 mon.a (mon.0) 995 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2023-07-29T17:21:26.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.750981+0000 mon.a (mon.0) 996 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2023-07-29T17:21:26.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.758542+0000 mon.a (mon.0) 997 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2023-07-29T17:21:26.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.694883+0000 mon.a (mon.0) 980 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.696787+0000 mon.a (mon.0) 981 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.698708+0000 mon.a (mon.0) 982 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.700791+0000 mon.a (mon.0) 983 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.702887+0000 mon.a (mon.0) 984 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.705727+0000 mon.a (mon.0) 985 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.708228+0000 mon.a (mon.0) 986 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.710814+0000 mon.a (mon.0) 987 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.713438+0000 mon.a (mon.0) 988 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.715618+0000 mon.a (mon.0) 989 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.717820+0000 mon.a (mon.0) 990 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:26.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: cephadm 2023-07-29T17:21:25.719062+0000 mgr.y (mgr.25169) 500 : cephadm 1 Upgrade: Finalizing container_image settings 2023-07-29T17:21:26.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.729146+0000 mon.a (mon.0) 991 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.730930+0000 mon.a (mon.0) 992 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2023-07-29T17:21:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.738527+0000 mon.a (mon.0) 993 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2023-07-29T17:21:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.740405+0000 mon.a (mon.0) 994 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.749212+0000 mon.a (mon.0) 995 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2023-07-29T17:21:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.750981+0000 mon.a (mon.0) 996 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2023-07-29T17:21:26.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.758542+0000 mon.a (mon.0) 997 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2023-07-29T17:21:26.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.760075+0000 mon.a (mon.0) 998 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2023-07-29T17:21:26.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.767967+0000 mon.a (mon.0) 999 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2023-07-29T17:21:26.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.769496+0000 mon.a (mon.0) 1000 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2023-07-29T17:21:26.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.778171+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2023-07-29T17:21:26.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.780057+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2023-07-29T17:21:26.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.787701+0000 mon.a (mon.0) 1003 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2023-07-29T17:21:26.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.789275+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2023-07-29T17:21:26.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.798888+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2023-07-29T17:21:26.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.800344+0000 mon.a (mon.0) 1006 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.801701+0000 mon.a (mon.0) 1007 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2023-07-29T17:21:26.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.811490+0000 mon.a (mon.0) 1008 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2023-07-29T17:21:26.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.813076+0000 mon.a (mon.0) 1009 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2023-07-29T17:21:26.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.821903+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2023-07-29T17:21:26.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.823332+0000 mon.a (mon.0) 1011 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2023-07-29T17:21:26.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.829642+0000 mon.a (mon.0) 1012 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2023-07-29T17:21:26.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.830809+0000 mon.a (mon.0) 1013 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.831817+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.760075+0000 mon.a (mon.0) 998 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.767967+0000 mon.a (mon.0) 999 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.769496+0000 mon.a (mon.0) 1000 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.778171+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.780057+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.787701+0000 mon.a (mon.0) 1003 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.789275+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2023-07-29T17:21:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.798888+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2023-07-29T17:21:26.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.800344+0000 mon.a (mon.0) 1006 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.801701+0000 mon.a (mon.0) 1007 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2023-07-29T17:21:26.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.811490+0000 mon.a (mon.0) 1008 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2023-07-29T17:21:26.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.813076+0000 mon.a (mon.0) 1009 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2023-07-29T17:21:26.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.821903+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2023-07-29T17:21:26.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.823332+0000 mon.a (mon.0) 1011 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2023-07-29T17:21:26.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.829642+0000 mon.a (mon.0) 1012 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2023-07-29T17:21:26.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.830809+0000 mon.a (mon.0) 1013 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.831817+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.833107+0000 mon.a (mon.0) 1015 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.834178+0000 mon.a (mon.0) 1016 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.835250+0000 mon.a (mon.0) 1017 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.836318+0000 mon.a (mon.0) 1018 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.837485+0000 mon.a (mon.0) 1019 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2023-07-29T17:21:26.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.842996+0000 mon.a (mon.0) 1020 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2023-07-29T17:21:26.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: cephadm 2023-07-29T17:21:25.844266+0000 mgr.y (mgr.25169) 501 : cephadm 1 Upgrade: Complete! 2023-07-29T17:21:26.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.845144+0000 mon.a (mon.0) 1021 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2023-07-29T17:21:26.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.849437+0000 mon.a (mon.0) 1022 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2023-07-29T17:21:26.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.851327+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.852650+0000 mon.a (mon.0) 1024 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: cephadm 2023-07-29T17:21:25.853860+0000 mgr.y (mgr.25169) 502 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.860321+0000 mon.a (mon.0) 1025 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.946576+0000 mon.a (mon.0) 1026 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.948263+0000 mon.a (mon.0) 1027 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: cephadm 2023-07-29T17:21:25.949456+0000 mgr.y (mgr.25169) 503 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[58614]: audit 2023-07-29T17:21:25.957822+0000 mon.a (mon.0) 1028 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.833107+0000 mon.a (mon.0) 1015 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.834178+0000 mon.a (mon.0) 1016 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.835250+0000 mon.a (mon.0) 1017 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.836318+0000 mon.a (mon.0) 1018 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-29T17:21:26.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.837485+0000 mon.a (mon.0) 1019 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2023-07-29T17:21:26.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.842996+0000 mon.a (mon.0) 1020 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2023-07-29T17:21:26.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: cephadm 2023-07-29T17:21:25.844266+0000 mgr.y (mgr.25169) 501 : cephadm 1 Upgrade: Complete! 2023-07-29T17:21:26.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.845144+0000 mon.a (mon.0) 1021 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2023-07-29T17:21:26.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.849437+0000 mon.a (mon.0) 1022 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2023-07-29T17:21:26.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.851327+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.852650+0000 mon.a (mon.0) 1024 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: cephadm 2023-07-29T17:21:25.853860+0000 mgr.y (mgr.25169) 502 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.860321+0000 mon.a (mon.0) 1025 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:26.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.946576+0000 mon.a (mon.0) 1026 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:21:26.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.948263+0000 mon.a (mon.0) 1027 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:21:26.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: cephadm 2023-07-29T17:21:25.949456+0000 mgr.y (mgr.25169) 503 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:21:26.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:26 smithi078 bash[60360]: audit 2023-07-29T17:21:25.957822+0000 mon.a (mon.0) 1028 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:27.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.828307884Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2023-07-29T17:21:27.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.878650945Z level=info msg="Executing migration" id="add annotations_enabled column" 2023-07-29T17:21:27.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.920735443Z level=info msg="Executing migration" id="add time_selection_enabled column" 2023-07-29T17:21:27.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.946061929Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2023-07-29T17:21:27.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.971338462Z level=info msg="Executing migration" id="add share column" 2023-07-29T17:21:27.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:26 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:26.996613586Z level=info msg="Executing migration" id="create default alerting folders" 2023-07-29T17:21:27.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.02192316Z level=info msg="Executing migration" id="create file table" 2023-07-29T17:21:27.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.063936781Z level=info msg="Executing migration" id="file table idx: path natural pk" 2023-07-29T17:21:27.355 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.139477097Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2023-07-29T17:21:27.355 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.165049058Z level=info msg="Executing migration" id="create file_meta table" 2023-07-29T17:21:27.355 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.206947393Z level=info msg="Executing migration" id="file table idx: path key" 2023-07-29T17:21:27.355 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.249115373Z level=info msg="Executing migration" id="set path collation in file table" 2023-07-29T17:21:27.356 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.291154415Z level=info msg="Executing migration" id="managed permissions migration" 2023-07-29T17:21:27.356 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.333184659Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2023-07-29T17:21:27.616 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.358441573Z level=info msg="Executing migration" id="RBAC action name migrator" 2023-07-29T17:21:27.616 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.383697221Z level=info msg="Executing migration" id="Add UID column to playlist" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.409076725Z level=info msg="Executing migration" id="Update uid column values in playlist" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.434418298Z level=info msg="Executing migration" id="Add index for uid in playlist" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.45959076Z level=info msg="Executing migration" id="update group index for alert rules" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.484909396Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.526977189Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.552270906Z level=info msg="Executing migration" id="add action column to seed_assignment" 2023-07-29T17:21:27.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.594273082Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2023-07-29T17:21:27.876 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.619545892Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2023-07-29T17:21:27.877 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.644819045Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2023-07-29T17:21:27.877 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.72855104Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2023-07-29T17:21:27.878 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.770528355Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2023-07-29T17:21:27.878 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.812586955Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2023-07-29T17:21:27.878 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.854625363Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2023-07-29T17:21:28.176 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.883451875Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2023-07-29T17:21:28.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.908760241Z level=info msg="Executing migration" id="create folder table" 2023-07-29T17:21:28.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.934006049Z level=info msg="Executing migration" id="Add index for parent_uid" 2023-07-29T17:21:28.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:27 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:27.984388723Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2023-07-29T17:21:28.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:28.009771474Z level=info msg="Executing migration" id="Update folder title length" 2023-07-29T17:21:28.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:28.051622452Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2023-07-29T17:21:28.177 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=migrator t=2023-07-29T17:21:28.076985211Z level=info msg="migrations completed" performed=485 skipped=0 duration=21.580320333s 2023-07-29T17:21:28.178 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=sqlstore t=2023-07-29T17:21:28.078377929Z level=info msg="Created default organization" 2023-07-29T17:21:28.676 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=plugin.loader t=2023-07-29T17:21:28.194588311Z level=info msg="Plugin registered" pluginID=input 2023-07-29T17:21:28.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=plugin.loader t=2023-07-29T17:21:28.223392183Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2023-07-29T17:21:28.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=plugin.loader t=2023-07-29T17:21:28.223417807Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2023-07-29T17:21:28.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=secrets t=2023-07-29T17:21:28.223456027Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2023-07-29T17:21:28.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=query_data t=2023-07-29T17:21:28.225322332Z level=info msg="Query Service initialization" 2023-07-29T17:21:28.677 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[74113]: logger=live.push_http t=2023-07-29T17:21:28.228618467Z level=info msg="Live Push Gateway initialization" 2023-07-29T17:21:28.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:28 smithi078 bash[58614]: cluster 2023-07-29T17:21:27.608904+0000 mgr.y (mgr.25169) 504 : cluster 0 pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:28.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:28 smithi078 bash[60360]: cluster 2023-07-29T17:21:27.608904+0000 mgr.y (mgr.25169) 504 : cluster 0 pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:29.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:28 smithi129 bash[53756]: cluster 2023-07-29T17:21:27.608904+0000 mgr.y (mgr.25169) 504 : cluster 0 pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:29.552 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=infra.usagestats.collector t=2023-07-29T17:21:29.187932794Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2023-07-29T17:21:29.552 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=provisioning.datasources t=2023-07-29T17:21:29.245285663Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2023-07-29T17:21:29.552 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=provisioning.datasources t=2023-07-29T17:21:29.339209325Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2023-07-29T17:21:29.552 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=provisioning.alerting t=2023-07-29T17:21:29.41325459Z level=info msg="starting to provision alerting" 2023-07-29T17:21:29.552 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=provisioning.alerting t=2023-07-29T17:21:29.413322812Z level=info msg="finished to provision alerting" 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=grafanaStorageLogger t=2023-07-29T17:21:29.413718601Z level=info msg="storage starting" 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=ngalert.state.manager t=2023-07-29T17:21:29.4137395Z level=info msg="Warming state cache for startup" 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=http.server t=2023-07-29T17:21:29.420008101Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=sqlstore.transactions t=2023-07-29T17:21:29.42540435Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=ngalert.state.manager t=2023-07-29T17:21:29.529410325Z level=info msg="State cache has been initialized" states=0 duration=115.667961ms 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=ngalert.multiorg.alertmanager t=2023-07-29T17:21:29.529541055Z level=info msg="starting MultiOrg Alertmanager" 2023-07-29T17:21:29.553 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=ticker t=2023-07-29T17:21:29.529619891Z level=info msg=starting first_tick=2023-07-29T17:21:30Z 2023-07-29T17:21:29.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[53756]: audit 2023-07-29T17:21:28.685154+0000 mon.a (mon.0) 1029 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:29.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=sqlstore.transactions t=2023-07-29T17:21:29.65166644Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-29T17:21:29.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=sqlstore.transactions t=2023-07-29T17:21:29.662734862Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2023-07-29T17:21:29.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=sqlstore.transactions t=2023-07-29T17:21:29.727473017Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2023-07-29T17:21:29.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=sqlstore.transactions t=2023-07-29T17:21:29.738874901Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2023-07-29T17:21:29.927 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:21:29 smithi129 bash[74113]: logger=sqlstore.transactions t=2023-07-29T17:21:29.749971053Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2023-07-29T17:21:30.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:29 smithi078 bash[58614]: audit 2023-07-29T17:21:28.685154+0000 mon.a (mon.0) 1029 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:30.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:29 smithi078 bash[60360]: audit 2023-07-29T17:21:28.685154+0000 mon.a (mon.0) 1029 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:21:31.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:30 smithi129 bash[53756]: cluster 2023-07-29T17:21:29.610194+0000 mgr.y (mgr.25169) 505 : cluster 0 pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:31.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:30 smithi129 bash[53756]: audit 2023-07-29T17:21:29.937043+0000 mgr.y (mgr.25169) 506 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:31.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:30 smithi078 bash[58614]: cluster 2023-07-29T17:21:29.610194+0000 mgr.y (mgr.25169) 505 : cluster 0 pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:31.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:30 smithi078 bash[58614]: audit 2023-07-29T17:21:29.937043+0000 mgr.y (mgr.25169) 506 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:31.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:30 smithi078 bash[60360]: cluster 2023-07-29T17:21:29.610194+0000 mgr.y (mgr.25169) 505 : cluster 0 pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:31.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:30 smithi078 bash[60360]: audit 2023-07-29T17:21:29.937043+0000 mgr.y (mgr.25169) 506 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:33.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:32 smithi129 bash[53756]: cluster 2023-07-29T17:21:31.611194+0000 mgr.y (mgr.25169) 507 : cluster 0 pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:33.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:32 smithi078 bash[58614]: cluster 2023-07-29T17:21:31.611194+0000 mgr.y (mgr.25169) 507 : cluster 0 pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:33.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:32 smithi078 bash[60360]: cluster 2023-07-29T17:21:31.611194+0000 mgr.y (mgr.25169) 507 : cluster 0 pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:35.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:34 smithi129 bash[53756]: cluster 2023-07-29T17:21:33.612782+0000 mgr.y (mgr.25169) 508 : cluster 0 pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:35.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:34 smithi078 bash[58614]: cluster 2023-07-29T17:21:33.612782+0000 mgr.y (mgr.25169) 508 : cluster 0 pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:35.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:34 smithi078 bash[60360]: cluster 2023-07-29T17:21:33.612782+0000 mgr.y (mgr.25169) 508 : cluster 0 pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:37.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:36 smithi129 bash[53756]: cluster 2023-07-29T17:21:35.613830+0000 mgr.y (mgr.25169) 509 : cluster 0 pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:37.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:36 smithi078 bash[58614]: cluster 2023-07-29T17:21:35.613830+0000 mgr.y (mgr.25169) 509 : cluster 0 pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:37.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:36 smithi078 bash[60360]: cluster 2023-07-29T17:21:35.613830+0000 mgr.y (mgr.25169) 509 : cluster 0 pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:39.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:38 smithi129 bash[53756]: cluster 2023-07-29T17:21:37.614930+0000 mgr.y (mgr.25169) 510 : cluster 0 pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:39.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:38 smithi078 bash[58614]: cluster 2023-07-29T17:21:37.614930+0000 mgr.y (mgr.25169) 510 : cluster 0 pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:39.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:38 smithi078 bash[60360]: cluster 2023-07-29T17:21:37.614930+0000 mgr.y (mgr.25169) 510 : cluster 0 pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:41.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:40 smithi129 bash[53756]: cluster 2023-07-29T17:21:39.616275+0000 mgr.y (mgr.25169) 511 : cluster 0 pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:41.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:40 smithi129 bash[53756]: audit 2023-07-29T17:21:39.948535+0000 mgr.y (mgr.25169) 512 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:41.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:40 smithi078 bash[58614]: cluster 2023-07-29T17:21:39.616275+0000 mgr.y (mgr.25169) 511 : cluster 0 pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:41.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:40 smithi078 bash[58614]: audit 2023-07-29T17:21:39.948535+0000 mgr.y (mgr.25169) 512 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:40 smithi078 bash[60360]: cluster 2023-07-29T17:21:39.616275+0000 mgr.y (mgr.25169) 511 : cluster 0 pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:41.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:40 smithi078 bash[60360]: audit 2023-07-29T17:21:39.948535+0000 mgr.y (mgr.25169) 512 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:43.176 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:42 smithi129 bash[53756]: cluster 2023-07-29T17:21:41.617233+0000 mgr.y (mgr.25169) 513 : cluster 0 pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:43.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:42 smithi078 bash[60360]: cluster 2023-07-29T17:21:41.617233+0000 mgr.y (mgr.25169) 513 : cluster 0 pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:43.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:42 smithi078 bash[58614]: cluster 2023-07-29T17:21:41.617233+0000 mgr.y (mgr.25169) 513 : cluster 0 pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:45.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:44 smithi129 bash[53756]: cluster 2023-07-29T17:21:43.618724+0000 mgr.y (mgr.25169) 514 : cluster 0 pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:45.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:44 smithi078 bash[58614]: cluster 2023-07-29T17:21:43.618724+0000 mgr.y (mgr.25169) 514 : cluster 0 pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:45.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:44 smithi078 bash[60360]: cluster 2023-07-29T17:21:43.618724+0000 mgr.y (mgr.25169) 514 : cluster 0 pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:47.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:46 smithi129 bash[53756]: cluster 2023-07-29T17:21:45.619802+0000 mgr.y (mgr.25169) 515 : cluster 0 pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:47.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:46 smithi078 bash[58614]: cluster 2023-07-29T17:21:45.619802+0000 mgr.y (mgr.25169) 515 : cluster 0 pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:47.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:46 smithi078 bash[60360]: cluster 2023-07-29T17:21:45.619802+0000 mgr.y (mgr.25169) 515 : cluster 0 pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:47.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch ps' 2023-07-29T17:21:48.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:47 smithi129 bash[53756]: audit 2023-07-29T17:21:46.412208+0000 mgr.y (mgr.25169) 516 : audit 0 from='client.35043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:48.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:47 smithi078 bash[58614]: audit 2023-07-29T17:21:46.412208+0000 mgr.y (mgr.25169) 516 : audit 0 from='client.35043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:48.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:47 smithi078 bash[60360]: audit 2023-07-29T17:21:46.412208+0000 mgr.y (mgr.25169) 516 : audit 0 from='client.35043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:49.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:48 smithi129 bash[53756]: cluster 2023-07-29T17:21:47.620958+0000 mgr.y (mgr.25169) 517 : cluster 0 pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:49.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:48 smithi078 bash[58614]: cluster 2023-07-29T17:21:47.620958+0000 mgr.y (mgr.25169) 517 : cluster 0 pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:49.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:48 smithi078 bash[60360]: cluster 2023-07-29T17:21:47.620958+0000 mgr.y (mgr.25169) 517 : cluster 0 pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:51.088 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (72s) 65s ago 17m 15.1M - 0.25.0 c8568f914cd2 575415f1371d 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (46s) 38s ago 17m 25.6M - 9.4.7 2c41d148cca3 360b40446815 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (2m) 38s ago 15m 47.1M - 3.6 b662b4d80c86 a64a43bc0b16 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (9m) 38s ago 22m 431M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (10m) 65s ago 24m 485M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (9m) 65s ago 24m 84.6M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:21:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (8m) 38s ago 22m 67.4M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (8m) 65s ago 22m 63.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2m) 65s ago 18m 8787k - 1.5.0 0da6a335fe13 4196bcd9b292 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (2m) 38s ago 18m 9944k - 1.5.0 0da6a335fe13 f9ecb2f5f5ff 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7m) 65s ago 21m 62.6M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6m) 65s ago 21m 65.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6m) 65s ago 21m 54.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5m) 65s ago 20m 62.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:21:51.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (5m) 38s ago 20m 62.7M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (4m) 38s ago 19m 56.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (4m) 38s ago 19m 54.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (3m) 38s ago 19m 56.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 running (99s) 38s ago 18m 38.0M - 2.43.0 a07b618ecd1d ce3cf859e4c4 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (3m) 65s ago 16m 84.3M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (3m) 38s ago 15m 84.6M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:21:51.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (3m) 65s ago 15m 84.5M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:21:51.092 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (2m) 38s ago 15m 84.6M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:21:51.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:50 smithi129 bash[53756]: cluster 2023-07-29T17:21:49.622314+0000 mgr.y (mgr.25169) 518 : cluster 0 pgmap v346: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:51.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:50 smithi129 bash[53756]: audit 2023-07-29T17:21:49.959964+0000 mgr.y (mgr.25169) 519 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:51.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:50 smithi078 bash[58614]: cluster 2023-07-29T17:21:49.622314+0000 mgr.y (mgr.25169) 518 : cluster 0 pgmap v346: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:51.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:50 smithi078 bash[58614]: audit 2023-07-29T17:21:49.959964+0000 mgr.y (mgr.25169) 519 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:51.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:50 smithi078 bash[60360]: cluster 2023-07-29T17:21:49.622314+0000 mgr.y (mgr.25169) 518 : cluster 0 pgmap v346: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:51.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:50 smithi078 bash[60360]: audit 2023-07-29T17:21:49.959964+0000 mgr.y (mgr.25169) 519 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:21:51.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph versions' 2023-07-29T17:21:52.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:51 smithi078 bash[58614]: audit 2023-07-29T17:21:51.041913+0000 mgr.y (mgr.25169) 520 : audit 0 from='client.35049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:52.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:51 smithi078 bash[60360]: audit 2023-07-29T17:21:51.041913+0000 mgr.y (mgr.25169) 520 : audit 0 from='client.35049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:52.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:51 smithi129 bash[53756]: audit 2023-07-29T17:21:51.041913+0000 mgr.y (mgr.25169) 520 : audit 0 from='client.35049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:21:53.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:52 smithi078 bash[58614]: cluster 2023-07-29T17:21:51.623219+0000 mgr.y (mgr.25169) 521 : cluster 0 pgmap v347: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:53.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:52 smithi078 bash[60360]: cluster 2023-07-29T17:21:51.623219+0000 mgr.y (mgr.25169) 521 : cluster 0 pgmap v347: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:53.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:52 smithi129 bash[53756]: cluster 2023-07-29T17:21:51.623219+0000 mgr.y (mgr.25169) 521 : cluster 0 pgmap v347: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:55.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:54 smithi078 bash[58614]: cluster 2023-07-29T17:21:53.624827+0000 mgr.y (mgr.25169) 522 : cluster 0 pgmap v348: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:55.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:54 smithi078 bash[60360]: cluster 2023-07-29T17:21:53.624827+0000 mgr.y (mgr.25169) 522 : cluster 0 pgmap v348: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:55.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:54 smithi129 bash[53756]: cluster 2023-07-29T17:21:53.624827+0000 mgr.y (mgr.25169) 522 : cluster 0 pgmap v348: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:21:55.654 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:21:55.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:21:55.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:21:55.656 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:21:56.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:55 smithi078 bash[58614]: audit 2023-07-29T17:21:55.628711+0000 mon.a (mon.0) 1030 : audit 0 from='client.44897 172.21.15.78:0/272973286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:56.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:55 smithi078 bash[60360]: audit 2023-07-29T17:21:55.628711+0000 mon.a (mon.0) 1030 : audit 0 from='client.44897 172.21.15.78:0/272973286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:56.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:55 smithi129 bash[53756]: audit 2023-07-29T17:21:55.628711+0000 mon.a (mon.0) 1030 : audit 0 from='client.44897 172.21.15.78:0/272973286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:21:56.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2023-07-29T17:21:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:56 smithi129 bash[53756]: cluster 2023-07-29T17:21:55.625816+0000 mgr.y (mgr.25169) 523 : cluster 0 pgmap v349: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:56 smithi078 bash[58614]: cluster 2023-07-29T17:21:55.625816+0000 mgr.y (mgr.25169) 523 : cluster 0 pgmap v349: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:57.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:56 smithi078 bash[60360]: cluster 2023-07-29T17:21:55.625816+0000 mgr.y (mgr.25169) 523 : cluster 0 pgmap v349: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:59.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:21:58 smithi078 bash[58614]: cluster 2023-07-29T17:21:57.627016+0000 mgr.y (mgr.25169) 524 : cluster 0 pgmap v350: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:59.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:21:58 smithi078 bash[60360]: cluster 2023-07-29T17:21:57.627016+0000 mgr.y (mgr.25169) 524 : cluster 0 pgmap v350: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:59.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:21:58 smithi129 bash[53756]: cluster 2023-07-29T17:21:57.627016+0000 mgr.y (mgr.25169) 524 : cluster 0 pgmap v350: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:21:59.771 INFO:teuthology.orchestra.run.smithi078.stdout:wait for servicemap items w/ changing names to refresh 2023-07-29T17:22:00.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'sleep 60' 2023-07-29T17:22:01.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:00 smithi129 bash[53756]: cluster 2023-07-29T17:21:59.628424+0000 mgr.y (mgr.25169) 525 : cluster 0 pgmap v351: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:01.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:00 smithi129 bash[53756]: audit 2023-07-29T17:21:59.969069+0000 mgr.y (mgr.25169) 526 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:01.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:00 smithi078 bash[60360]: cluster 2023-07-29T17:21:59.628424+0000 mgr.y (mgr.25169) 525 : cluster 0 pgmap v351: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:01.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:00 smithi078 bash[60360]: audit 2023-07-29T17:21:59.969069+0000 mgr.y (mgr.25169) 526 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:01.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:00 smithi078 bash[58614]: cluster 2023-07-29T17:21:59.628424+0000 mgr.y (mgr.25169) 525 : cluster 0 pgmap v351: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:01.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:00 smithi078 bash[58614]: audit 2023-07-29T17:21:59.969069+0000 mgr.y (mgr.25169) 526 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:03.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:02 smithi078 bash[60360]: cluster 2023-07-29T17:22:01.629391+0000 mgr.y (mgr.25169) 527 : cluster 0 pgmap v352: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:03.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:02 smithi078 bash[58614]: cluster 2023-07-29T17:22:01.629391+0000 mgr.y (mgr.25169) 527 : cluster 0 pgmap v352: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:03.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:02 smithi129 bash[53756]: cluster 2023-07-29T17:22:01.629391+0000 mgr.y (mgr.25169) 527 : cluster 0 pgmap v352: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:05.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:04 smithi129 bash[53756]: cluster 2023-07-29T17:22:03.630997+0000 mgr.y (mgr.25169) 528 : cluster 0 pgmap v353: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:05.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:04 smithi078 bash[58614]: cluster 2023-07-29T17:22:03.630997+0000 mgr.y (mgr.25169) 528 : cluster 0 pgmap v353: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:05.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:04 smithi078 bash[60360]: cluster 2023-07-29T17:22:03.630997+0000 mgr.y (mgr.25169) 528 : cluster 0 pgmap v353: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:07.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:06 smithi129 bash[53756]: cluster 2023-07-29T17:22:05.632198+0000 mgr.y (mgr.25169) 529 : cluster 0 pgmap v354: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:07.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:06 smithi078 bash[60360]: cluster 2023-07-29T17:22:05.632198+0000 mgr.y (mgr.25169) 529 : cluster 0 pgmap v354: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:07.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:06 smithi078 bash[58614]: cluster 2023-07-29T17:22:05.632198+0000 mgr.y (mgr.25169) 529 : cluster 0 pgmap v354: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:09.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:08 smithi129 bash[53756]: cluster 2023-07-29T17:22:07.633367+0000 mgr.y (mgr.25169) 530 : cluster 0 pgmap v355: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:09.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:08 smithi078 bash[58614]: cluster 2023-07-29T17:22:07.633367+0000 mgr.y (mgr.25169) 530 : cluster 0 pgmap v355: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:09.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:08 smithi078 bash[60360]: cluster 2023-07-29T17:22:07.633367+0000 mgr.y (mgr.25169) 530 : cluster 0 pgmap v355: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:11.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:10 smithi129 bash[53756]: cluster 2023-07-29T17:22:09.634615+0000 mgr.y (mgr.25169) 531 : cluster 0 pgmap v356: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:11.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:10 smithi129 bash[53756]: audit 2023-07-29T17:22:09.979717+0000 mgr.y (mgr.25169) 532 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:11.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:10 smithi078 bash[58614]: cluster 2023-07-29T17:22:09.634615+0000 mgr.y (mgr.25169) 531 : cluster 0 pgmap v356: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:11.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:10 smithi078 bash[58614]: audit 2023-07-29T17:22:09.979717+0000 mgr.y (mgr.25169) 532 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:11.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:10 smithi078 bash[60360]: cluster 2023-07-29T17:22:09.634615+0000 mgr.y (mgr.25169) 531 : cluster 0 pgmap v356: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:11.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:10 smithi078 bash[60360]: audit 2023-07-29T17:22:09.979717+0000 mgr.y (mgr.25169) 532 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:13.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:12 smithi129 bash[53756]: cluster 2023-07-29T17:22:11.635562+0000 mgr.y (mgr.25169) 533 : cluster 0 pgmap v357: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:13.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:12 smithi078 bash[58614]: cluster 2023-07-29T17:22:11.635562+0000 mgr.y (mgr.25169) 533 : cluster 0 pgmap v357: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:13.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:12 smithi078 bash[60360]: cluster 2023-07-29T17:22:11.635562+0000 mgr.y (mgr.25169) 533 : cluster 0 pgmap v357: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:15.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:14 smithi129 bash[53756]: cluster 2023-07-29T17:22:13.637027+0000 mgr.y (mgr.25169) 534 : cluster 0 pgmap v358: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:15.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:14 smithi078 bash[58614]: cluster 2023-07-29T17:22:13.637027+0000 mgr.y (mgr.25169) 534 : cluster 0 pgmap v358: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:15.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:14 smithi078 bash[60360]: cluster 2023-07-29T17:22:13.637027+0000 mgr.y (mgr.25169) 534 : cluster 0 pgmap v358: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:17.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:16 smithi129 bash[53756]: cluster 2023-07-29T17:22:15.638074+0000 mgr.y (mgr.25169) 535 : cluster 0 pgmap v359: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:17.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:16 smithi078 bash[60360]: cluster 2023-07-29T17:22:15.638074+0000 mgr.y (mgr.25169) 535 : cluster 0 pgmap v359: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:17.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:16 smithi078 bash[58614]: cluster 2023-07-29T17:22:15.638074+0000 mgr.y (mgr.25169) 535 : cluster 0 pgmap v359: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:19.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:18 smithi129 bash[53756]: cluster 2023-07-29T17:22:17.639215+0000 mgr.y (mgr.25169) 536 : cluster 0 pgmap v360: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:19.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:18 smithi078 bash[58614]: cluster 2023-07-29T17:22:17.639215+0000 mgr.y (mgr.25169) 536 : cluster 0 pgmap v360: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:18 smithi078 bash[60360]: cluster 2023-07-29T17:22:17.639215+0000 mgr.y (mgr.25169) 536 : cluster 0 pgmap v360: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:21.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:20 smithi129 bash[53756]: cluster 2023-07-29T17:22:19.640537+0000 mgr.y (mgr.25169) 537 : cluster 0 pgmap v361: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:21.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:20 smithi129 bash[53756]: audit 2023-07-29T17:22:19.988877+0000 mgr.y (mgr.25169) 538 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:21.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:20 smithi078 bash[58614]: cluster 2023-07-29T17:22:19.640537+0000 mgr.y (mgr.25169) 537 : cluster 0 pgmap v361: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:21.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:20 smithi078 bash[58614]: audit 2023-07-29T17:22:19.988877+0000 mgr.y (mgr.25169) 538 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:21.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:20 smithi078 bash[60360]: cluster 2023-07-29T17:22:19.640537+0000 mgr.y (mgr.25169) 537 : cluster 0 pgmap v361: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:21.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:20 smithi078 bash[60360]: audit 2023-07-29T17:22:19.988877+0000 mgr.y (mgr.25169) 538 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:23.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:22 smithi129 bash[53756]: cluster 2023-07-29T17:22:21.641455+0000 mgr.y (mgr.25169) 539 : cluster 0 pgmap v362: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:23.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:22 smithi078 bash[58614]: cluster 2023-07-29T17:22:21.641455+0000 mgr.y (mgr.25169) 539 : cluster 0 pgmap v362: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:23.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:22 smithi078 bash[60360]: cluster 2023-07-29T17:22:21.641455+0000 mgr.y (mgr.25169) 539 : cluster 0 pgmap v362: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:25.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:24 smithi129 bash[53756]: cluster 2023-07-29T17:22:23.642946+0000 mgr.y (mgr.25169) 540 : cluster 0 pgmap v363: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:25.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:24 smithi078 bash[58614]: cluster 2023-07-29T17:22:23.642946+0000 mgr.y (mgr.25169) 540 : cluster 0 pgmap v363: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:25.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:24 smithi078 bash[60360]: cluster 2023-07-29T17:22:23.642946+0000 mgr.y (mgr.25169) 540 : cluster 0 pgmap v363: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:27.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:27 smithi129 bash[53756]: cluster 2023-07-29T17:22:25.644020+0000 mgr.y (mgr.25169) 541 : cluster 0 pgmap v364: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:27.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:27 smithi129 bash[53756]: audit 2023-07-29T17:22:26.575941+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:22:27.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:27 smithi129 bash[53756]: audit 2023-07-29T17:22:26.577612+0000 mon.a (mon.0) 1032 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:22:27.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:27 smithi129 bash[53756]: audit 2023-07-29T17:22:26.586498+0000 mon.a (mon.0) 1033 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:22:27.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:26 smithi078 bash[58614]: cluster 2023-07-29T17:22:25.644020+0000 mgr.y (mgr.25169) 541 : cluster 0 pgmap v364: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:27.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:26 smithi078 bash[58614]: audit 2023-07-29T17:22:26.575941+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:22:27.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:26 smithi078 bash[58614]: audit 2023-07-29T17:22:26.577612+0000 mon.a (mon.0) 1032 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:22:27.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:26 smithi078 bash[58614]: audit 2023-07-29T17:22:26.586498+0000 mon.a (mon.0) 1033 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:22:27.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:27 smithi078 bash[60360]: cluster 2023-07-29T17:22:25.644020+0000 mgr.y (mgr.25169) 541 : cluster 0 pgmap v364: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:27.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:27 smithi078 bash[60360]: audit 2023-07-29T17:22:26.575941+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:22:27.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:27 smithi078 bash[60360]: audit 2023-07-29T17:22:26.577612+0000 mon.a (mon.0) 1032 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:22:27.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:27 smithi078 bash[60360]: audit 2023-07-29T17:22:26.586498+0000 mon.a (mon.0) 1033 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:22:28.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:28 smithi129 bash[53756]: cephadm 2023-07-29T17:22:26.579045+0000 mgr.y (mgr.25169) 542 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:22:28.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:28 smithi078 bash[58614]: cephadm 2023-07-29T17:22:26.579045+0000 mgr.y (mgr.25169) 542 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:22:28.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:28 smithi078 bash[60360]: cephadm 2023-07-29T17:22:26.579045+0000 mgr.y (mgr.25169) 542 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:22:29.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:29 smithi129 bash[53756]: cluster 2023-07-29T17:22:27.645159+0000 mgr.y (mgr.25169) 543 : cluster 0 pgmap v365: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:29.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:29 smithi078 bash[58614]: cluster 2023-07-29T17:22:27.645159+0000 mgr.y (mgr.25169) 543 : cluster 0 pgmap v365: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:29.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:29 smithi078 bash[60360]: cluster 2023-07-29T17:22:27.645159+0000 mgr.y (mgr.25169) 543 : cluster 0 pgmap v365: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:31.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:31 smithi129 bash[53756]: cluster 2023-07-29T17:22:29.646494+0000 mgr.y (mgr.25169) 544 : cluster 0 pgmap v366: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:31.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:31 smithi129 bash[53756]: audit 2023-07-29T17:22:29.995331+0000 mgr.y (mgr.25169) 545 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:31 smithi078 bash[58614]: cluster 2023-07-29T17:22:29.646494+0000 mgr.y (mgr.25169) 544 : cluster 0 pgmap v366: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:31.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:31 smithi078 bash[58614]: audit 2023-07-29T17:22:29.995331+0000 mgr.y (mgr.25169) 545 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:31 smithi078 bash[60360]: cluster 2023-07-29T17:22:29.646494+0000 mgr.y (mgr.25169) 544 : cluster 0 pgmap v366: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:31 smithi078 bash[60360]: audit 2023-07-29T17:22:29.995331+0000 mgr.y (mgr.25169) 545 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:33.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:33 smithi129 bash[53756]: cluster 2023-07-29T17:22:31.647469+0000 mgr.y (mgr.25169) 546 : cluster 0 pgmap v367: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:33.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:33 smithi078 bash[58614]: cluster 2023-07-29T17:22:31.647469+0000 mgr.y (mgr.25169) 546 : cluster 0 pgmap v367: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:33.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:33 smithi078 bash[60360]: cluster 2023-07-29T17:22:31.647469+0000 mgr.y (mgr.25169) 546 : cluster 0 pgmap v367: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:35.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:35 smithi129 bash[53756]: cluster 2023-07-29T17:22:33.648986+0000 mgr.y (mgr.25169) 547 : cluster 0 pgmap v368: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:35.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:35 smithi078 bash[58614]: cluster 2023-07-29T17:22:33.648986+0000 mgr.y (mgr.25169) 547 : cluster 0 pgmap v368: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:35.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:35 smithi078 bash[60360]: cluster 2023-07-29T17:22:33.648986+0000 mgr.y (mgr.25169) 547 : cluster 0 pgmap v368: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:37.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:37 smithi129 bash[53756]: cluster 2023-07-29T17:22:35.650034+0000 mgr.y (mgr.25169) 548 : cluster 0 pgmap v369: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:37.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:37 smithi078 bash[58614]: cluster 2023-07-29T17:22:35.650034+0000 mgr.y (mgr.25169) 548 : cluster 0 pgmap v369: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:37.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:37 smithi078 bash[60360]: cluster 2023-07-29T17:22:35.650034+0000 mgr.y (mgr.25169) 548 : cluster 0 pgmap v369: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:39.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:39 smithi129 bash[53756]: cluster 2023-07-29T17:22:37.651145+0000 mgr.y (mgr.25169) 549 : cluster 0 pgmap v370: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:39.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:39 smithi078 bash[58614]: cluster 2023-07-29T17:22:37.651145+0000 mgr.y (mgr.25169) 549 : cluster 0 pgmap v370: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:39.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:39 smithi078 bash[60360]: cluster 2023-07-29T17:22:37.651145+0000 mgr.y (mgr.25169) 549 : cluster 0 pgmap v370: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:41.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:41 smithi129 bash[53756]: cluster 2023-07-29T17:22:39.652515+0000 mgr.y (mgr.25169) 550 : cluster 0 pgmap v371: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:41.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:41 smithi129 bash[53756]: audit 2023-07-29T17:22:40.004777+0000 mgr.y (mgr.25169) 551 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:41.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:41 smithi078 bash[58614]: cluster 2023-07-29T17:22:39.652515+0000 mgr.y (mgr.25169) 550 : cluster 0 pgmap v371: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:41.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:41 smithi078 bash[58614]: audit 2023-07-29T17:22:40.004777+0000 mgr.y (mgr.25169) 551 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:41.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:41 smithi078 bash[60360]: cluster 2023-07-29T17:22:39.652515+0000 mgr.y (mgr.25169) 550 : cluster 0 pgmap v371: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:41.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:41 smithi078 bash[60360]: audit 2023-07-29T17:22:40.004777+0000 mgr.y (mgr.25169) 551 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:43.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:43 smithi129 bash[53756]: cluster 2023-07-29T17:22:41.653398+0000 mgr.y (mgr.25169) 552 : cluster 0 pgmap v372: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:43.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:43 smithi078 bash[58614]: cluster 2023-07-29T17:22:41.653398+0000 mgr.y (mgr.25169) 552 : cluster 0 pgmap v372: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:43.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:43 smithi078 bash[60360]: cluster 2023-07-29T17:22:41.653398+0000 mgr.y (mgr.25169) 552 : cluster 0 pgmap v372: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:45.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:45 smithi129 bash[53756]: cluster 2023-07-29T17:22:43.654878+0000 mgr.y (mgr.25169) 553 : cluster 0 pgmap v373: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:45.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:45 smithi078 bash[58614]: cluster 2023-07-29T17:22:43.654878+0000 mgr.y (mgr.25169) 553 : cluster 0 pgmap v373: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:45.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:45 smithi078 bash[60360]: cluster 2023-07-29T17:22:43.654878+0000 mgr.y (mgr.25169) 553 : cluster 0 pgmap v373: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:47.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:47 smithi129 bash[53756]: cluster 2023-07-29T17:22:45.655930+0000 mgr.y (mgr.25169) 554 : cluster 0 pgmap v374: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:47.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:47 smithi078 bash[58614]: cluster 2023-07-29T17:22:45.655930+0000 mgr.y (mgr.25169) 554 : cluster 0 pgmap v374: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:47.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:47 smithi078 bash[60360]: cluster 2023-07-29T17:22:45.655930+0000 mgr.y (mgr.25169) 554 : cluster 0 pgmap v374: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:49.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:49 smithi129 bash[53756]: cluster 2023-07-29T17:22:47.657017+0000 mgr.y (mgr.25169) 555 : cluster 0 pgmap v375: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:49.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:49 smithi078 bash[58614]: cluster 2023-07-29T17:22:47.657017+0000 mgr.y (mgr.25169) 555 : cluster 0 pgmap v375: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:49.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:49 smithi078 bash[60360]: cluster 2023-07-29T17:22:47.657017+0000 mgr.y (mgr.25169) 555 : cluster 0 pgmap v375: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:51.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:51 smithi129 bash[53756]: cluster 2023-07-29T17:22:49.658292+0000 mgr.y (mgr.25169) 556 : cluster 0 pgmap v376: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:51.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:51 smithi129 bash[53756]: audit 2023-07-29T17:22:50.016279+0000 mgr.y (mgr.25169) 557 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:51.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:51 smithi078 bash[58614]: cluster 2023-07-29T17:22:49.658292+0000 mgr.y (mgr.25169) 556 : cluster 0 pgmap v376: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:51.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:51 smithi078 bash[58614]: audit 2023-07-29T17:22:50.016279+0000 mgr.y (mgr.25169) 557 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:51.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:51 smithi078 bash[60360]: cluster 2023-07-29T17:22:49.658292+0000 mgr.y (mgr.25169) 556 : cluster 0 pgmap v376: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:51.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:51 smithi078 bash[60360]: audit 2023-07-29T17:22:50.016279+0000 mgr.y (mgr.25169) 557 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:22:53.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:53 smithi129 bash[53756]: cluster 2023-07-29T17:22:51.659158+0000 mgr.y (mgr.25169) 558 : cluster 0 pgmap v377: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:53.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:53 smithi078 bash[60360]: cluster 2023-07-29T17:22:51.659158+0000 mgr.y (mgr.25169) 558 : cluster 0 pgmap v377: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:53.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:53 smithi078 bash[58614]: cluster 2023-07-29T17:22:51.659158+0000 mgr.y (mgr.25169) 558 : cluster 0 pgmap v377: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:55.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:55 smithi129 bash[53756]: cluster 2023-07-29T17:22:53.660753+0000 mgr.y (mgr.25169) 559 : cluster 0 pgmap v378: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:55.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:55 smithi078 bash[58614]: cluster 2023-07-29T17:22:53.660753+0000 mgr.y (mgr.25169) 559 : cluster 0 pgmap v378: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:55.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:55 smithi078 bash[60360]: cluster 2023-07-29T17:22:53.660753+0000 mgr.y (mgr.25169) 559 : cluster 0 pgmap v378: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:22:57.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:57 smithi129 bash[53756]: cluster 2023-07-29T17:22:55.661797+0000 mgr.y (mgr.25169) 560 : cluster 0 pgmap v379: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:57.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:57 smithi078 bash[58614]: cluster 2023-07-29T17:22:55.661797+0000 mgr.y (mgr.25169) 560 : cluster 0 pgmap v379: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:57.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:57 smithi078 bash[60360]: cluster 2023-07-29T17:22:55.661797+0000 mgr.y (mgr.25169) 560 : cluster 0 pgmap v379: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:59.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:22:59 smithi129 bash[53756]: cluster 2023-07-29T17:22:57.662955+0000 mgr.y (mgr.25169) 561 : cluster 0 pgmap v380: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:59.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:22:59 smithi078 bash[58614]: cluster 2023-07-29T17:22:57.662955+0000 mgr.y (mgr.25169) 561 : cluster 0 pgmap v380: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:22:59.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:22:59 smithi078 bash[60360]: cluster 2023-07-29T17:22:57.662955+0000 mgr.y (mgr.25169) 561 : cluster 0 pgmap v380: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:01.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:01 smithi129 bash[53756]: cluster 2023-07-29T17:22:59.664268+0000 mgr.y (mgr.25169) 562 : cluster 0 pgmap v381: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:01.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:01 smithi129 bash[53756]: audit 2023-07-29T17:23:00.027718+0000 mgr.y (mgr.25169) 563 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:01.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:01 smithi078 bash[58614]: cluster 2023-07-29T17:22:59.664268+0000 mgr.y (mgr.25169) 562 : cluster 0 pgmap v381: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:01.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:01 smithi078 bash[58614]: audit 2023-07-29T17:23:00.027718+0000 mgr.y (mgr.25169) 563 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:01.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:01 smithi078 bash[60360]: cluster 2023-07-29T17:22:59.664268+0000 mgr.y (mgr.25169) 562 : cluster 0 pgmap v381: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:01.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:01 smithi078 bash[60360]: audit 2023-07-29T17:23:00.027718+0000 mgr.y (mgr.25169) 563 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:03.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:03 smithi129 bash[53756]: cluster 2023-07-29T17:23:01.665188+0000 mgr.y (mgr.25169) 564 : cluster 0 pgmap v382: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:03.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:03 smithi078 bash[58614]: cluster 2023-07-29T17:23:01.665188+0000 mgr.y (mgr.25169) 564 : cluster 0 pgmap v382: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:03.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:03 smithi078 bash[60360]: cluster 2023-07-29T17:23:01.665188+0000 mgr.y (mgr.25169) 564 : cluster 0 pgmap v382: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:04.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch ps' 2023-07-29T17:23:05.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:05 smithi129 bash[53756]: cluster 2023-07-29T17:23:03.666758+0000 mgr.y (mgr.25169) 565 : cluster 0 pgmap v383: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:05.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:05 smithi078 bash[58614]: cluster 2023-07-29T17:23:03.666758+0000 mgr.y (mgr.25169) 565 : cluster 0 pgmap v383: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:05.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:05 smithi078 bash[60360]: cluster 2023-07-29T17:23:03.666758+0000 mgr.y (mgr.25169) 565 : cluster 0 pgmap v383: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:07.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:07 smithi129 bash[53756]: cluster 2023-07-29T17:23:05.667893+0000 mgr.y (mgr.25169) 566 : cluster 0 pgmap v384: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:07.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:07 smithi078 bash[58614]: cluster 2023-07-29T17:23:05.667893+0000 mgr.y (mgr.25169) 566 : cluster 0 pgmap v384: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:07.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:07 smithi078 bash[60360]: cluster 2023-07-29T17:23:05.667893+0000 mgr.y (mgr.25169) 566 : cluster 0 pgmap v384: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:08.123 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T17:23:08.123 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2m) 2m ago 19m 15.1M - 0.25.0 c8568f914cd2 575415f1371d 2023-07-29T17:23:08.123 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi129 *:3000 running (2m) 115s ago 18m 25.6M - 9.4.7 2c41d148cca3 360b40446815 2023-07-29T17:23:08.123 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi129.vbvpri smithi129 running (3m) 115s ago 16m 47.1M - 3.6 b662b4d80c86 a64a43bc0b16 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi129 *:8443,8765 running (11m) 115s ago 23m 431M - 18.1.3-11-g0484097b b662b4d80c86 b1f88d573752 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:8443,8765 running (12m) 2m ago 25m 485M - 18.1.3-11-g0484097b b662b4d80c86 755ab2cf9b41 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (10m) 2m ago 25m 84.6M 2048M 18.1.3-11-g0484097b b662b4d80c86 4dbba3830709 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi129 running (9m) 115s ago 23m 67.4M 2048M 18.1.3-11-g0484097b b662b4d80c86 3aea258ae30d 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 2m ago 23m 63.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 42c5c18ad6ee 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3m) 2m ago 19m 8787k - 1.5.0 0da6a335fe13 4196bcd9b292 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi129 *:9100 running (3m) 115s ago 19m 9944k - 1.5.0 0da6a335fe13 f9ecb2f5f5ff 2023-07-29T17:23:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (8m) 2m ago 23m 62.6M 4096M 18.1.3-11-g0484097b b662b4d80c86 01abbd015470 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7m) 2m ago 22m 65.5M 4096M 18.1.3-11-g0484097b b662b4d80c86 7402038a807d 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7m) 2m ago 22m 54.3M 4096M 18.1.3-11-g0484097b b662b4d80c86 bee91c737d13 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6m) 2m ago 21m 62.0M 4096M 18.1.3-11-g0484097b b662b4d80c86 fb874ed16889 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi129 running (6m) 115s ago 21m 62.7M 4096M 18.1.3-11-g0484097b b662b4d80c86 060c2b154aeb 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi129 running (5m) 115s ago 21m 56.2M 4096M 18.1.3-11-g0484097b b662b4d80c86 cc1e01846b2e 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi129 running (5m) 115s ago 20m 54.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 045bf9c19080 2023-07-29T17:23:08.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi129 running (4m) 115s ago 20m 56.8M 4096M 18.1.3-11-g0484097b b662b4d80c86 77c4b29859df 2023-07-29T17:23:08.126 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi129 *:9095 running (2m) 115s ago 19m 38.0M - 2.43.0 a07b618ecd1d ce3cf859e4c4 2023-07-29T17:23:08.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.lydpyb smithi078 *:8000 running (4m) 2m ago 17m 84.3M - 18.1.3-11-g0484097b b662b4d80c86 e8036e2cae6a 2023-07-29T17:23:08.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi129.dbexjf smithi129 *:8000 running (4m) 115s ago 17m 84.6M - 18.1.3-11-g0484097b b662b4d80c86 903be7c085ee 2023-07-29T17:23:08.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.xbxtjr smithi078 *:80 running (4m) 2m ago 17m 84.5M - 18.1.3-11-g0484097b b662b4d80c86 d6b799d034a1 2023-07-29T17:23:08.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi129.tqsnpp smithi129 *:80 running (4m) 115s ago 17m 84.6M - 18.1.3-11-g0484097b b662b4d80c86 215bbe49c995 2023-07-29T17:23:08.969 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph versions' 2023-07-29T17:23:09.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:09 smithi129 bash[53756]: cluster 2023-07-29T17:23:07.668998+0000 mgr.y (mgr.25169) 567 : cluster 0 pgmap v385: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:09.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:09 smithi129 bash[53756]: audit 2023-07-29T17:23:08.074335+0000 mgr.y (mgr.25169) 568 : audit 0 from='client.44903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:09.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:09 smithi078 bash[58614]: cluster 2023-07-29T17:23:07.668998+0000 mgr.y (mgr.25169) 567 : cluster 0 pgmap v385: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:09.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:09 smithi078 bash[58614]: audit 2023-07-29T17:23:08.074335+0000 mgr.y (mgr.25169) 568 : audit 0 from='client.44903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:09.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:09 smithi078 bash[60360]: cluster 2023-07-29T17:23:07.668998+0000 mgr.y (mgr.25169) 567 : cluster 0 pgmap v385: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:09.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:09 smithi078 bash[60360]: audit 2023-07-29T17:23:08.074335+0000 mgr.y (mgr.25169) 568 : audit 0 from='client.44903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:11.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:11 smithi129 bash[53756]: cluster 2023-07-29T17:23:09.670244+0000 mgr.y (mgr.25169) 569 : cluster 0 pgmap v386: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:11.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:11 smithi129 bash[53756]: audit 2023-07-29T17:23:10.033536+0000 mgr.y (mgr.25169) 570 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:11.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:11 smithi078 bash[58614]: cluster 2023-07-29T17:23:09.670244+0000 mgr.y (mgr.25169) 569 : cluster 0 pgmap v386: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:11.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:11 smithi078 bash[58614]: audit 2023-07-29T17:23:10.033536+0000 mgr.y (mgr.25169) 570 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:11.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:11 smithi078 bash[60360]: cluster 2023-07-29T17:23:09.670244+0000 mgr.y (mgr.25169) 569 : cluster 0 pgmap v386: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:11.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:11 smithi078 bash[60360]: audit 2023-07-29T17:23:10.033536+0000 mgr.y (mgr.25169) 570 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:23:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2023-07-29T17:23:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 3 2023-07-29T17:23:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:23:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2023-07-29T17:23:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 2 2023-07-29T17:23:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:23:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2023-07-29T17:23:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 8 2023-07-29T17:23:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 4 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)": 17 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-29T17:23:12.620 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:23:13.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:13 smithi078 bash[58614]: cluster 2023-07-29T17:23:11.671170+0000 mgr.y (mgr.25169) 571 : cluster 0 pgmap v387: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:13.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:13 smithi078 bash[58614]: audit 2023-07-29T17:23:12.591877+0000 mon.a (mon.0) 1034 : audit 0 from='client.44909 172.21.15.78:0/2285913717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:13.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:13 smithi078 bash[60360]: cluster 2023-07-29T17:23:11.671170+0000 mgr.y (mgr.25169) 571 : cluster 0 pgmap v387: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:13.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:13 smithi078 bash[60360]: audit 2023-07-29T17:23:12.591877+0000 mon.a (mon.0) 1034 : audit 0 from='client.44909 172.21.15.78:0/2285913717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:13.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch upgrade status' 2023-07-29T17:23:13.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:13 smithi129 bash[53756]: cluster 2023-07-29T17:23:11.671170+0000 mgr.y (mgr.25169) 571 : cluster 0 pgmap v387: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:13.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:13 smithi129 bash[53756]: audit 2023-07-29T17:23:12.591877+0000 mon.a (mon.0) 1034 : audit 0 from='client.44909 172.21.15.78:0/2285913717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:15.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:15 smithi129 bash[53756]: cluster 2023-07-29T17:23:13.672749+0000 mgr.y (mgr.25169) 572 : cluster 0 pgmap v388: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:15.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:15 smithi078 bash[58614]: cluster 2023-07-29T17:23:13.672749+0000 mgr.y (mgr.25169) 572 : cluster 0 pgmap v388: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:15.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:15 smithi078 bash[60360]: cluster 2023-07-29T17:23:13.672749+0000 mgr.y (mgr.25169) 572 : cluster 0 pgmap v388: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:17.008 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-29T17:23:17.009 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": null, 2023-07-29T17:23:17.009 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": false, 2023-07-29T17:23:17.009 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "", 2023-07-29T17:23:17.009 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2023-07-29T17:23:17.010 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": null, 2023-07-29T17:23:17.010 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "", 2023-07-29T17:23:17.010 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2023-07-29T17:23:17.011 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-29T17:23:17.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:17 smithi129 bash[53756]: cluster 2023-07-29T17:23:15.673627+0000 mgr.y (mgr.25169) 573 : cluster 0 pgmap v389: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:17.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:17 smithi078 bash[58614]: cluster 2023-07-29T17:23:15.673627+0000 mgr.y (mgr.25169) 573 : cluster 0 pgmap v389: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:17.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:17 smithi078 bash[60360]: cluster 2023-07-29T17:23:15.673627+0000 mgr.y (mgr.25169) 573 : cluster 0 pgmap v389: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:17.788 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph health detail' 2023-07-29T17:23:18.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:18 smithi129 bash[53756]: audit 2023-07-29T17:23:16.982120+0000 mgr.y (mgr.25169) 574 : audit 0 from='client.44915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:18.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:18 smithi078 bash[58614]: audit 2023-07-29T17:23:16.982120+0000 mgr.y (mgr.25169) 574 : audit 0 from='client.44915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:18.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:18 smithi078 bash[60360]: audit 2023-07-29T17:23:16.982120+0000 mgr.y (mgr.25169) 574 : audit 0 from='client.44915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:19.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:19 smithi129 bash[53756]: cluster 2023-07-29T17:23:17.674717+0000 mgr.y (mgr.25169) 575 : cluster 0 pgmap v390: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:19.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:19 smithi078 bash[58614]: cluster 2023-07-29T17:23:17.674717+0000 mgr.y (mgr.25169) 575 : cluster 0 pgmap v390: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:19.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:19 smithi078 bash[60360]: cluster 2023-07-29T17:23:17.674717+0000 mgr.y (mgr.25169) 575 : cluster 0 pgmap v390: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:21.427 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:21 smithi129 bash[53756]: cluster 2023-07-29T17:23:19.676136+0000 mgr.y (mgr.25169) 576 : cluster 0 pgmap v391: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:21.428 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:21 smithi129 bash[53756]: audit 2023-07-29T17:23:20.044846+0000 mgr.y (mgr.25169) 577 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:21.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:21 smithi078 bash[58614]: cluster 2023-07-29T17:23:19.676136+0000 mgr.y (mgr.25169) 576 : cluster 0 pgmap v391: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:21.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:21 smithi078 bash[58614]: audit 2023-07-29T17:23:20.044846+0000 mgr.y (mgr.25169) 577 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:21.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:21 smithi078 bash[60360]: cluster 2023-07-29T17:23:19.676136+0000 mgr.y (mgr.25169) 576 : cluster 0 pgmap v391: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:21.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:21 smithi078 bash[60360]: audit 2023-07-29T17:23:20.044846+0000 mgr.y (mgr.25169) 577 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:21.592 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_OK 2023-07-29T17:23:22.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:22 smithi078 bash[58614]: audit 2023-07-29T17:23:21.566646+0000 mon.c (mon.2) 18 : audit 0 from='client.35079 172.21.15.78:0/2285314267' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:23:22.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:22 smithi078 bash[60360]: audit 2023-07-29T17:23:21.566646+0000 mon.c (mon.2) 18 : audit 0 from='client.35079 172.21.15.78:0/2285314267' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:23:22.437 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2023-07-29T17:23:22.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:22 smithi129 bash[53756]: audit 2023-07-29T17:23:21.566646+0000 mon.c (mon.2) 18 : audit 0 from='client.35079 172.21.15.78:0/2285314267' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-29T17:23:23.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:23 smithi078 bash[58614]: cluster 2023-07-29T17:23:21.677049+0000 mgr.y (mgr.25169) 578 : cluster 0 pgmap v392: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:23.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:23 smithi078 bash[60360]: cluster 2023-07-29T17:23:21.677049+0000 mgr.y (mgr.25169) 578 : cluster 0 pgmap v392: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:23.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:23 smithi129 bash[53756]: cluster 2023-07-29T17:23:21.677049+0000 mgr.y (mgr.25169) 578 : cluster 0 pgmap v392: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:25.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:25 smithi078 bash[58614]: cluster 2023-07-29T17:23:23.678537+0000 mgr.y (mgr.25169) 579 : cluster 0 pgmap v393: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:25.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:25 smithi078 bash[60360]: cluster 2023-07-29T17:23:23.678537+0000 mgr.y (mgr.25169) 579 : cluster 0 pgmap v393: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:25.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:25 smithi129 bash[53756]: cluster 2023-07-29T17:23:23.678537+0000 mgr.y (mgr.25169) 579 : cluster 0 pgmap v393: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:26.348 INFO:teuthology.orchestra.run.smithi078.stdout:true 2023-07-29T17:23:27.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2023-07-29T17:23:27.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:27 smithi078 bash[58614]: cluster 2023-07-29T17:23:25.679551+0000 mgr.y (mgr.25169) 580 : cluster 0 pgmap v394: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:27.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:27 smithi078 bash[58614]: audit 2023-07-29T17:23:26.307448+0000 mon.a (mon.0) 1035 : audit 0 from='client.44927 172.21.15.78:0/1143672420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:27.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:27 smithi078 bash[60360]: cluster 2023-07-29T17:23:25.679551+0000 mgr.y (mgr.25169) 580 : cluster 0 pgmap v394: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:27.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:27 smithi078 bash[60360]: audit 2023-07-29T17:23:26.307448+0000 mon.a (mon.0) 1035 : audit 0 from='client.44927 172.21.15.78:0/1143672420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:27.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:27 smithi129 bash[53756]: cluster 2023-07-29T17:23:25.679551+0000 mgr.y (mgr.25169) 580 : cluster 0 pgmap v394: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:27.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:27 smithi129 bash[53756]: audit 2023-07-29T17:23:26.307448+0000 mon.a (mon.0) 1035 : audit 0 from='client.44927 172.21.15.78:0/1143672420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:28.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[58614]: audit 2023-07-29T17:23:27.201836+0000 mon.a (mon.0) 1036 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:23:28.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[58614]: audit 2023-07-29T17:23:27.203487+0000 mon.a (mon.0) 1037 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:23:28.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[58614]: cephadm 2023-07-29T17:23:27.204724+0000 mgr.y (mgr.25169) 581 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:23:28.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[58614]: audit 2023-07-29T17:23:27.212568+0000 mon.a (mon.0) 1038 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:23:28.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[60360]: audit 2023-07-29T17:23:27.201836+0000 mon.a (mon.0) 1036 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:23:28.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[60360]: audit 2023-07-29T17:23:27.203487+0000 mon.a (mon.0) 1037 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:23:28.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[60360]: cephadm 2023-07-29T17:23:27.204724+0000 mgr.y (mgr.25169) 581 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:23:28.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:28 smithi078 bash[60360]: audit 2023-07-29T17:23:27.212568+0000 mon.a (mon.0) 1038 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:23:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:28 smithi129 bash[53756]: audit 2023-07-29T17:23:27.201836+0000 mon.a (mon.0) 1036 : audit 0 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T17:23:28.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:28 smithi129 bash[53756]: audit 2023-07-29T17:23:27.203487+0000 mon.a (mon.0) 1037 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T17:23:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:28 smithi129 bash[53756]: cephadm 2023-07-29T17:23:27.204724+0000 mgr.y (mgr.25169) 581 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-29T17:23:28.679 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:28 smithi129 bash[53756]: audit 2023-07-29T17:23:27.212568+0000 mon.a (mon.0) 1038 : audit 1 from='mgr.25169 172.21.15.78:0/1309315249' entity='mgr.y' 2023-07-29T17:23:29.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:29 smithi129 bash[53756]: cluster 2023-07-29T17:23:27.680621+0000 mgr.y (mgr.25169) 582 : cluster 0 pgmap v395: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:29.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:29 smithi078 bash[58614]: cluster 2023-07-29T17:23:27.680621+0000 mgr.y (mgr.25169) 582 : cluster 0 pgmap v395: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:29.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:29 smithi078 bash[60360]: cluster 2023-07-29T17:23:27.680621+0000 mgr.y (mgr.25169) 582 : cluster 0 pgmap v395: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)" 2023-07-29T17:23:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:31 smithi078 bash[60360]: cluster 2023-07-29T17:23:29.682031+0000 mgr.y (mgr.25169) 583 : cluster 0 pgmap v396: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:31.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:31 smithi078 bash[60360]: audit 2023-07-29T17:23:30.052612+0000 mgr.y (mgr.25169) 584 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:31.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:31 smithi078 bash[60360]: audit 2023-07-29T17:23:30.820331+0000 mon.c (mon.2) 19 : audit 0 from='client.44933 172.21.15.78:0/3782765668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:31.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:31 smithi078 bash[58614]: cluster 2023-07-29T17:23:29.682031+0000 mgr.y (mgr.25169) 583 : cluster 0 pgmap v396: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:31.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:31 smithi078 bash[58614]: audit 2023-07-29T17:23:30.052612+0000 mgr.y (mgr.25169) 584 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:31.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:31 smithi078 bash[58614]: audit 2023-07-29T17:23:30.820331+0000 mon.c (mon.2) 19 : audit 0 from='client.44933 172.21.15.78:0/3782765668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:31.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -e sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2023-07-29T17:23:31.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:31 smithi129 bash[53756]: cluster 2023-07-29T17:23:29.682031+0000 mgr.y (mgr.25169) 583 : cluster 0 pgmap v396: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:31.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:31 smithi129 bash[53756]: audit 2023-07-29T17:23:30.052612+0000 mgr.y (mgr.25169) 584 : audit 0 from='client.54580 -' entity='client.iscsi.foo.smithi129.vbvpri' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-29T17:23:31.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:31 smithi129 bash[53756]: audit 2023-07-29T17:23:30.820331+0000 mon.c (mon.2) 19 : audit 0 from='client.44933 172.21.15.78:0/3782765668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-29T17:23:33.678 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:33 smithi129 bash[53756]: cluster 2023-07-29T17:23:31.683063+0000 mgr.y (mgr.25169) 585 : cluster 0 pgmap v397: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:33.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:33 smithi078 bash[58614]: cluster 2023-07-29T17:23:31.683063+0000 mgr.y (mgr.25169) 585 : cluster 0 pgmap v397: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:33.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:33 smithi078 bash[60360]: cluster 2023-07-29T17:23:31.683063+0000 mgr.y (mgr.25169) 585 : cluster 0 pgmap v397: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-29T17:23:35.308 INFO:teuthology.orchestra.run.smithi078.stdout:osd 8 2m ago - 2023-07-29T17:23:35.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:35 smithi129 bash[53756]: cluster 2023-07-29T17:23:33.684643+0000 mgr.y (mgr.25169) 586 : cluster 0 pgmap v398: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:35.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:35 smithi078 bash[58614]: cluster 2023-07-29T17:23:33.684643+0000 mgr.y (mgr.25169) 586 : cluster 0 pgmap v398: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:35.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:35 smithi078 bash[60360]: cluster 2023-07-29T17:23:33.684643+0000 mgr.y (mgr.25169) 586 : cluster 0 pgmap v398: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-29T17:23:36.136 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-29T17:23:36.150 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-29T17:23:36.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- bash -c 'ceph orch upgrade ls' 2023-07-29T17:23:36.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:36 smithi078 bash[58614]: audit 2023-07-29T17:23:35.256694+0000 mgr.y (mgr.25169) 587 : audit 0 from='client.44939 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:36.448 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:36 smithi078 bash[60360]: audit 2023-07-29T17:23:35.256694+0000 mgr.y (mgr.25169) 587 : audit 0 from='client.44939 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:36.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:36 smithi129 bash[53756]: audit 2023-07-29T17:23:35.256694+0000 mgr.y (mgr.25169) 587 : audit 0 from='client.44939 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T17:23:37.656 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:37 smithi078 bash[60360]: cluster 2023-07-29T17:23:35.685709+0000 mgr.y (mgr.25169) 588 : cluster 0 pgmap v399: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:37.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:37 smithi078 bash[58614]: cluster 2023-07-29T17:23:35.685709+0000 mgr.y (mgr.25169) 588 : cluster 0 pgmap v399: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:37.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:37 smithi129 bash[53756]: cluster 2023-07-29T17:23:35.685709+0000 mgr.y (mgr.25169) 588 : cluster 0 pgmap v399: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:39.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:39 smithi129 bash[53756]: cluster 2023-07-29T17:23:37.686832+0000 mgr.y (mgr.25169) 589 : cluster 0 pgmap v400: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:39.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[58614]: cluster 2023-07-29T17:23:37.686832+0000 mgr.y (mgr.25169) 589 : cluster 0 pgmap v400: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:39.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[60360]: cluster 2023-07-29T17:23:37.686832+0000 mgr.y (mgr.25169) 589 : cluster 0 pgmap v400: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-29T17:23:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: Traceback (most recent call last): 2023-07-29T17:23:39.929 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1774, in _handle_command 2023-07-29T17:23:39.929 INFO:teuthology.orchestra.run.smithi078.stderr: return self.handle_command(inbuf, cmd) 2023-07-29T17:23:39.929 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command 2023-07-29T17:23:39.929 INFO:teuthology.orchestra.run.smithi078.stderr: return dispatch[cmd['prefix']].call(self, cmd, inbuf) 2023-07-29T17:23:39.929 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: return self.func(mgr, **kwargs) 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: return func(*args, **kwargs) 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1708, in _upgrade_ls 2023-07-29T17:23:39.930 INFO:teuthology.orchestra.run.smithi078.stderr: r = raise_if_exception(completion) 2023-07-29T17:23:39.931 INFO:teuthology.orchestra.run.smithi078.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception 2023-07-29T17:23:39.931 INFO:teuthology.orchestra.run.smithi078.stderr: raise e 2023-07-29T17:23:39.931 INFO:teuthology.orchestra.run.smithi078.stderr:requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-29T17:23:39.931 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-29T17:23:40.198 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: debug 2023-07-29T17:23:39.825+0000 7f8fc2999700 -1 log_channel(cephadm) log [ERR] : 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-29T17:23:40.198 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: Traceback (most recent call last): 2023-07-29T17:23:40.199 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper 2023-07-29T17:23:40.199 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: return OrchResult(f(*args, **kwargs)) 2023-07-29T17:23:40.199 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3184, in upgrade_ls 2023-07-29T17:23:40.199 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: return self.upgrade.upgrade_ls(image, tags, show_all_versions) 2023-07-29T17:23:40.200 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 280, in upgrade_ls 2023-07-29T17:23:40.200 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: ls = reg.get_tags(bare_image) 2023-07-29T17:23:40.200 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/cephadm/registry.py", line 50, in get_tags 2023-07-29T17:23:40.201 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: token = self.get_token(r) 2023-07-29T17:23:40.201 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/cephadm/registry.py", line 20, in get_token 2023-07-29T17:23:40.201 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: r.raise_for_status() 2023-07-29T17:23:40.201 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status 2023-07-29T17:23:40.202 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: raise HTTPError(http_error_msg, response=self) 2023-07-29T17:23:40.202 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-29T17:23:40.202 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: debug 2023-07-29T17:23:39.893+0000 7f8fc2999700 -1 mgr handle_command module 'orchestrator' command handler threw exception: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-29T17:23:40.203 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: debug 2023-07-29T17:23:39.893+0000 7f8fc2999700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last): 2023-07-29T17:23:40.203 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/mgr_module.py", line 1774, in _handle_command 2023-07-29T17:23:40.203 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: return self.handle_command(inbuf, cmd) 2023-07-29T17:23:40.203 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command 2023-07-29T17:23:40.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: return dispatch[cmd['prefix']].call(self, cmd, inbuf) 2023-07-29T17:23:40.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call 2023-07-29T17:23:40.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: return self.func(mgr, **kwargs) 2023-07-29T17:23:40.205 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 2023-07-29T17:23:40.205 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 2023-07-29T17:23:40.205 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper 2023-07-29T17:23:40.205 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: return func(*args, **kwargs) 2023-07-29T17:23:40.206 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1708, in _upgrade_ls 2023-07-29T17:23:40.206 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: r = raise_if_exception(completion) 2023-07-29T17:23:40.206 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception 2023-07-29T17:23:40.206 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: raise e 2023-07-29T17:23:40.206 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:39 smithi078 bash[55873]: requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-29T17:23:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-29T17:23:40.778 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/cephadm.py", line 1230, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi078 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-07-29T17:23:40.849 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f0613ce4061c4998a7abb954b5cd5691 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/cephadm.py", line 1230, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi078 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-07-29T17:23:40.852 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-29T17:23:40.862 INFO:tasks.cephadm:Teardown begin 2023-07-29T17:23:40.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T17:23:40.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T17:23:40.926 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-29T17:23:40.927 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-29T17:23:40.958 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-29T17:23:40.975 INFO:tasks.cephadm:Stopping all daemons... 2023-07-29T17:23:40.975 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-29T17:23:40.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a 2023-07-29T17:23:41.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:40 smithi078 systemd[1]: Stopping Ceph mon.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:41.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:41 smithi078 bash[58614]: debug 2023-07-29T17:23:41.161+0000 7f5945246700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T17:23:41.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 29 17:23:41 smithi078 bash[58614]: debug 2023-07-29T17:23:41.161+0000 7f5945246700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2023-07-29T17:23:42.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.a.service' 2023-07-29T17:23:42.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:23:42.160 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-29T17:23:42.160 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2023-07-29T17:23:42.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.c 2023-07-29T17:23:42.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:42 smithi078 systemd[1]: Stopping Ceph mon.c for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:42.599 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:42 smithi078 bash[60360]: debug 2023-07-29T17:23:42.305+0000 7f0cc8ae1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T17:23:42.599 INFO:journalctl@ceph.mon.c.smithi078.stdout:Jul 29 17:23:42 smithi078 bash[60360]: debug 2023-07-29T17:23:42.305+0000 7f0cc8ae1700 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2023-07-29T17:23:43.398 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.c.service' 2023-07-29T17:23:43.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:23:43.432 INFO:tasks.cephadm.mon.b:Stopped mon.c 2023-07-29T17:23:43.432 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2023-07-29T17:23:43.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.b 2023-07-29T17:23:43.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:43 smithi129 systemd[1]: Stopping Ceph mon.b for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:43.812 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:43 smithi129 bash[53756]: debug 2023-07-29T17:23:43.608+0000 7f5537a87700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-29T17:23:43.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Jul 29 17:23:43 smithi129 bash[53756]: debug 2023-07-29T17:23:43.608+0000 7f5537a87700 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2023-07-29T17:23:44.681 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mon.b.service' 2023-07-29T17:23:44.726 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:23:44.727 INFO:tasks.cephadm.mon.b:Stopped mon.b 2023-07-29T17:23:44.727 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-29T17:23:44.728 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y 2023-07-29T17:23:44.991 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:44 smithi078 systemd[1]: Stopping Ceph mgr.y for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:44.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:44 smithi078 bash[55873]: debug 2023-07-29T17:23:44.909+0000 7f903b0ee700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:23:44.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Jul 29 17:23:44 smithi078 bash[55873]: debug 2023-07-29T17:23:44.909+0000 7f903b0ee700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-29T17:23:45.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.y.service' 2023-07-29T17:23:45.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:23:45.838 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-29T17:23:45.839 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-29T17:23:45.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x 2023-07-29T17:23:46.177 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Jul 29 17:23:45 smithi129 systemd[1]: Stopping Ceph mgr.x for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:46.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@mgr.x.service' 2023-07-29T17:23:46.931 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:23:46.932 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-29T17:23:46.932 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-29T17:23:46.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.0 2023-07-29T17:23:47.197 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:23:46 smithi078 systemd[1]: Stopping Ceph osd.0 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:47.198 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:23:47 smithi078 bash[63468]: debug 2023-07-29T17:23:47.109+0000 7ff504577700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:23:47.198 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:23:47 smithi078 bash[63468]: debug 2023-07-29T17:23:47.109+0000 7ff504577700 -1 osd.0 125 *** Got signal Terminated *** 2023-07-29T17:23:47.198 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:23:47 smithi078 bash[63468]: debug 2023-07-29T17:23:47.109+0000 7ff504577700 -1 osd.0 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:23:53.697 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 29 17:23:53 smithi078 bash[83184]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-0 2023-07-29T17:23:55.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.0.service' 2023-07-29T17:23:55.786 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:23:55.786 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-29T17:23:55.787 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-29T17:23:55.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.1 2023-07-29T17:23:56.026 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:23:55 smithi078 systemd[1]: Stopping Ceph osd.1 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:23:56.447 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:23:56 smithi078 bash[65820]: debug 2023-07-29T17:23:55.997+0000 7fc89517f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:23:56.447 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:23:56 smithi078 bash[65820]: debug 2023-07-29T17:23:55.997+0000 7fc89517f700 -1 osd.1 125 *** Got signal Terminated *** 2023-07-29T17:23:56.448 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:23:56 smithi078 bash[65820]: debug 2023-07-29T17:23:55.997+0000 7fc89517f700 -1 osd.1 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:02.447 INFO:journalctl@ceph.osd.1.smithi078.stdout:Jul 29 17:24:02 smithi078 bash[83412]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-1 2023-07-29T17:24:03.697 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:03 smithi078 bash[78981]: ts=2023-07-29T17:24:03.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:03.698 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:03 smithi078 bash[78981]: ts=2023-07-29T17:24:03.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:04.507 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.1.service' 2023-07-29T17:24:04.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:04.542 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-29T17:24:04.542 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-29T17:24:04.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.2 2023-07-29T17:24:04.948 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:24:04 smithi078 systemd[1]: Stopping Ceph osd.2 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:04.948 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:24:04 smithi078 bash[68132]: debug 2023-07-29T17:24:04.669+0000 7fd6f81d3700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:24:04.948 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:24:04 smithi078 bash[68132]: debug 2023-07-29T17:24:04.669+0000 7fd6f81d3700 -1 osd.2 125 *** Got signal Terminated *** 2023-07-29T17:24:04.949 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:24:04 smithi078 bash[68132]: debug 2023-07-29T17:24:04.669+0000 7fd6f81d3700 -1 osd.2 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:11.447 INFO:journalctl@ceph.osd.2.smithi078.stdout:Jul 29 17:24:10 smithi078 bash[83643]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-2 2023-07-29T17:24:13.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.2.service' 2023-07-29T17:24:13.110 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:13.111 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-29T17:24:13.111 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-29T17:24:13.112 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.3 2023-07-29T17:24:13.178 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:12 smithi129 bash[71906]: ts=2023-07-29T17:24:12.688Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.78:8765: connect: connection refused" 2023-07-29T17:24:13.178 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:12 smithi129 bash[71906]: ts=2023-07-29T17:24:12.697Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.78:8765: connect: connection refused" 2023-07-29T17:24:13.178 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:12 smithi129 bash[71906]: ts=2023-07-29T17:24:12.697Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.78:8765: connect: connection refused" 2023-07-29T17:24:13.179 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:12 smithi129 bash[71906]: ts=2023-07-29T17:24:12.697Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.78:8765: connect: connection refused" 2023-07-29T17:24:13.322 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:13 smithi078 systemd[1]: Stopping Ceph osd.3 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:13.697 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:13 smithi078 bash[70437]: debug 2023-07-29T17:24:13.293+0000 7fc8a46c6700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:24:13.698 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:13 smithi078 bash[70437]: debug 2023-07-29T17:24:13.293+0000 7fc8a46c6700 -1 osd.3 125 *** Got signal Terminated *** 2023-07-29T17:24:13.698 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:13 smithi078 bash[70437]: debug 2023-07-29T17:24:13.293+0000 7fc8a46c6700 -1 osd.3 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:13.698 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:13 smithi078 bash[78981]: ts=2023-07-29T17:24:13.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:13.698 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:13 smithi078 bash[78981]: ts=2023-07-29T17:24:13.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:13.699 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:13 smithi078 bash[78981]: ts=2023-07-29T17:24:13.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:14.928 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:14 smithi129 bash[58650]: debug 2023-07-29T17:24:14.596+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:15.640 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:15 smithi129 bash[56428]: debug 2023-07-29T17:24:15.268+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:15.928 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:15 smithi129 bash[58650]: debug 2023-07-29T17:24:15.608+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:16.197 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:15 smithi078 bash[70437]: debug 2023-07-29T17:24:15.829+0000 7fc89ecb8700 -1 osd.3 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.875240+0000 front 2023-07-29T17:23:50.874862+0000 (oldest deadline 2023-07-29T17:24:15.575059+0000) 2023-07-29T17:24:16.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:16 smithi129 bash[56428]: debug 2023-07-29T17:24:16.304+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:17.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:16 smithi129 bash[58650]: debug 2023-07-29T17:24:16.656+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:17.197 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:16 smithi078 bash[70437]: debug 2023-07-29T17:24:16.789+0000 7fc89ecb8700 -1 osd.3 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.875240+0000 front 2023-07-29T17:23:50.874862+0000 (oldest deadline 2023-07-29T17:24:15.575059+0000) 2023-07-29T17:24:17.514 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:17 smithi129 bash[56428]: debug 2023-07-29T17:24:17.272+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:17.515 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:17 smithi129 bash[60866]: debug 2023-07-29T17:24:17.172+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:17.928 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:17 smithi129 bash[58650]: debug 2023-07-29T17:24:17.680+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:17.928 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:17 smithi129 bash[63075]: debug 2023-07-29T17:24:17.484+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:18.197 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:17 smithi078 bash[70437]: debug 2023-07-29T17:24:17.793+0000 7fc89ecb8700 -1 osd.3 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.875240+0000 front 2023-07-29T17:23:50.874862+0000 (oldest deadline 2023-07-29T17:24:15.575059+0000) 2023-07-29T17:24:18.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:18 smithi129 bash[56428]: debug 2023-07-29T17:24:18.284+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:18.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:18 smithi129 bash[60866]: debug 2023-07-29T17:24:18.196+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:18.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:18 smithi129 bash[63075]: debug 2023-07-29T17:24:18.444+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:19.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:18 smithi129 bash[58650]: debug 2023-07-29T17:24:18.688+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:19.477 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:19 smithi129 bash[60866]: debug 2023-07-29T17:24:19.176+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:19.478 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:19 smithi129 bash[56428]: debug 2023-07-29T17:24:19.284+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:19.928 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:19 smithi129 bash[58650]: debug 2023-07-29T17:24:19.684+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:19.929 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:19 smithi129 bash[63075]: debug 2023-07-29T17:24:19.448+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:19.947 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 29 17:24:19 smithi078 bash[83872]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-3 2023-07-29T17:24:20.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:20 smithi129 bash[63075]: debug 2023-07-29T17:24:20.404+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:20.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:20 smithi129 bash[56428]: debug 2023-07-29T17:24:20.272+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:20.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:20 smithi129 bash[60866]: debug 2023-07-29T17:24:20.208+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:21.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:20 smithi129 bash[58650]: debug 2023-07-29T17:24:20.696+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:21.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.3.service' 2023-07-29T17:24:21.584 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:21.584 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-29T17:24:21.585 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-29T17:24:21.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.4 2023-07-29T17:24:21.598 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[60866]: debug 2023-07-29T17:24:21.188+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:21.598 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[56428]: debug 2023-07-29T17:24:21.300+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:21.599 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[63075]: debug 2023-07-29T17:24:21.360+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:21.928 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[58650]: debug 2023-07-29T17:24:21.728+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:21.928 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:21 smithi129 systemd[1]: Stopping Ceph osd.4 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:21.929 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[56428]: debug 2023-07-29T17:24:21.760+0000 7fbca1d90700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:24:21.929 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[56428]: debug 2023-07-29T17:24:21.760+0000 7fbca1d90700 -1 osd.4 125 *** Got signal Terminated *** 2023-07-29T17:24:21.930 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:21 smithi129 bash[56428]: debug 2023-07-29T17:24:21.760+0000 7fbca1d90700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:22.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:22 smithi129 bash[56428]: debug 2023-07-29T17:24:22.300+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:22.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:22 smithi129 bash[60866]: debug 2023-07-29T17:24:22.228+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:22.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:22 smithi129 bash[63075]: debug 2023-07-29T17:24:22.328+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:23.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:22 smithi129 bash[58650]: debug 2023-07-29T17:24:22.732+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:23.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[60866]: debug 2023-07-29T17:24:23.224+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:23.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[56428]: debug 2023-07-29T17:24:23.276+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:23.679 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[56428]: debug 2023-07-29T17:24:23.276+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:58.807190+0000 front 2023-07-29T17:23:58.807167+0000 (oldest deadline 2023-07-29T17:24:22.306950+0000) 2023-07-29T17:24:23.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[63075]: debug 2023-07-29T17:24:23.328+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:23.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[63075]: debug 2023-07-29T17:24:23.328+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:23.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:23 smithi078 bash[78981]: ts=2023-07-29T17:24:23.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:23.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:23 smithi078 bash[78981]: ts=2023-07-29T17:24:23.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:23.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:23 smithi078 bash[78981]: ts=2023-07-29T17:24:23.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:24.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[58650]: debug 2023-07-29T17:24:23.776+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:24.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:23 smithi129 bash[58650]: debug 2023-07-29T17:24:23.776+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:24.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[60866]: debug 2023-07-29T17:24:24.220+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:24.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[60866]: debug 2023-07-29T17:24:24.220+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:24.679 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[56428]: debug 2023-07-29T17:24:24.264+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:24.679 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[56428]: debug 2023-07-29T17:24:24.264+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:58.807190+0000 front 2023-07-29T17:23:58.807167+0000 (oldest deadline 2023-07-29T17:24:22.306950+0000) 2023-07-29T17:24:24.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[63075]: debug 2023-07-29T17:24:24.332+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:24.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[63075]: debug 2023-07-29T17:24:24.332+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:25.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[58650]: debug 2023-07-29T17:24:24.808+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:25.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:24 smithi129 bash[58650]: debug 2023-07-29T17:24:24.808+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:25.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[56428]: debug 2023-07-29T17:24:25.316+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:25.678 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[56428]: debug 2023-07-29T17:24:25.316+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:58.807190+0000 front 2023-07-29T17:23:58.807167+0000 (oldest deadline 2023-07-29T17:24:22.306950+0000) 2023-07-29T17:24:25.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[60866]: debug 2023-07-29T17:24:25.256+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:25.680 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[60866]: debug 2023-07-29T17:24:25.256+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:25.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[63075]: debug 2023-07-29T17:24:25.376+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:25.681 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[63075]: debug 2023-07-29T17:24:25.376+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:26.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[58650]: debug 2023-07-29T17:24:25.804+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:26.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:25 smithi129 bash[58650]: debug 2023-07-29T17:24:25.804+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:26.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[60866]: debug 2023-07-29T17:24:26.220+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:26.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[60866]: debug 2023-07-29T17:24:26.220+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:26.680 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[56428]: debug 2023-07-29T17:24:26.360+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:47.104731+0000 front 2023-07-29T17:23:47.105043+0000 (oldest deadline 2023-07-29T17:24:14.705473+0000) 2023-07-29T17:24:26.680 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[56428]: debug 2023-07-29T17:24:26.360+0000 7fbc9c382700 -1 osd.4 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:58.807190+0000 front 2023-07-29T17:23:58.807167+0000 (oldest deadline 2023-07-29T17:24:22.306950+0000) 2023-07-29T17:24:26.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[63075]: debug 2023-07-29T17:24:26.396+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:26.681 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[63075]: debug 2023-07-29T17:24:26.396+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:27.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[58650]: debug 2023-07-29T17:24:26.848+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:27.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:26 smithi129 bash[58650]: debug 2023-07-29T17:24:26.848+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:27.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:27 smithi129 bash[60866]: debug 2023-07-29T17:24:27.232+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:27.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:27 smithi129 bash[60866]: debug 2023-07-29T17:24:27.232+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:27.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:27 smithi129 bash[63075]: debug 2023-07-29T17:24:27.388+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:27.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:27 smithi129 bash[63075]: debug 2023-07-29T17:24:27.388+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:28.178 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[75808]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-4 2023-07-29T17:24:28.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:27 smithi129 bash[58650]: debug 2023-07-29T17:24:27.804+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:28.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:27 smithi129 bash[58650]: debug 2023-07-29T17:24:27.804+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:28.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[60866]: debug 2023-07-29T17:24:28.243+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:28.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[60866]: debug 2023-07-29T17:24:28.243+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:28.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[63075]: debug 2023-07-29T17:24:28.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:28.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[63075]: debug 2023-07-29T17:24:28.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:29.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[58650]: debug 2023-07-29T17:24:28.755+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:29.179 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:28 smithi129 bash[58650]: debug 2023-07-29T17:24:28.755+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:29.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:29 smithi129 bash[60866]: debug 2023-07-29T17:24:29.259+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:29.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:29 smithi129 bash[60866]: debug 2023-07-29T17:24:29.259+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:29.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:29 smithi129 bash[63075]: debug 2023-07-29T17:24:29.315+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:29.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:29 smithi129 bash[63075]: debug 2023-07-29T17:24:29.315+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:30.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:29 smithi129 bash[58650]: debug 2023-07-29T17:24:29.755+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:30.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:29 smithi129 bash[58650]: debug 2023-07-29T17:24:29.755+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:30.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.4.service' 2023-07-29T17:24:30.530 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:30 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.4.service: Succeeded. 2023-07-29T17:24:30.530 INFO:journalctl@ceph.osd.4.smithi129.stdout:Jul 29 17:24:30 smithi129 systemd[1]: Stopped Ceph osd.4 for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:24:30.531 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[63075]: debug 2023-07-29T17:24:30.287+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:30.531 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[63075]: debug 2023-07-29T17:24:30.287+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:30.531 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[60866]: debug 2023-07-29T17:24:30.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:30.532 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[60866]: debug 2023-07-29T17:24:30.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:30.554 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:30.554 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-29T17:24:30.554 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-29T17:24:30.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.5 2023-07-29T17:24:30.788 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:30 smithi129 systemd[1]: Stopping Ceph osd.5 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:30.788 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[58650]: debug 2023-07-29T17:24:30.699+0000 7f7b24d2a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:24:30.789 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[58650]: debug 2023-07-29T17:24:30.699+0000 7f7b24d2a700 -1 osd.5 125 *** Got signal Terminated *** 2023-07-29T17:24:30.789 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[58650]: debug 2023-07-29T17:24:30.699+0000 7f7b24d2a700 -1 osd.5 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:31.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[58650]: debug 2023-07-29T17:24:30.759+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:31.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:30 smithi129 bash[58650]: debug 2023-07-29T17:24:30.759+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:31.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:31 smithi129 bash[60866]: debug 2023-07-29T17:24:31.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:31.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:31 smithi129 bash[60866]: debug 2023-07-29T17:24:31.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:31.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:31 smithi129 bash[63075]: debug 2023-07-29T17:24:31.319+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:31.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:31 smithi129 bash[63075]: debug 2023-07-29T17:24:31.319+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:32.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:31 smithi129 bash[58650]: debug 2023-07-29T17:24:31.799+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:32.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:31 smithi129 bash[58650]: debug 2023-07-29T17:24:31.799+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:32.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[60866]: debug 2023-07-29T17:24:32.199+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:32.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[60866]: debug 2023-07-29T17:24:32.199+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:32.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[63075]: debug 2023-07-29T17:24:32.315+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:32.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[63075]: debug 2023-07-29T17:24:32.315+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:33.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[58650]: debug 2023-07-29T17:24:32.771+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:33.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[58650]: debug 2023-07-29T17:24:32.771+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:33.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:32 smithi129 bash[58650]: debug 2023-07-29T17:24:32.771+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.516530+0000 front 2023-07-29T17:24:08.516559+0000 (oldest deadline 2023-07-29T17:24:32.016225+0000) 2023-07-29T17:24:33.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[60866]: debug 2023-07-29T17:24:33.175+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:33.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[60866]: debug 2023-07-29T17:24:33.175+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:33.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[63075]: debug 2023-07-29T17:24:33.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:33.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[63075]: debug 2023-07-29T17:24:33.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:33.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[63075]: debug 2023-07-29T17:24:33.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:33.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:33 smithi078 bash[78981]: ts=2023-07-29T17:24:33.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:33.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:33 smithi078 bash[78981]: ts=2023-07-29T17:24:33.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:33.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:33 smithi078 bash[78981]: ts=2023-07-29T17:24:33.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:34.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[58650]: debug 2023-07-29T17:24:33.815+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:34.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[58650]: debug 2023-07-29T17:24:33.815+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:34.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:33 smithi129 bash[58650]: debug 2023-07-29T17:24:33.815+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.516530+0000 front 2023-07-29T17:24:08.516559+0000 (oldest deadline 2023-07-29T17:24:32.016225+0000) 2023-07-29T17:24:34.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[60866]: debug 2023-07-29T17:24:34.179+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:34.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[60866]: debug 2023-07-29T17:24:34.179+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:34.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[63075]: debug 2023-07-29T17:24:34.347+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:34.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[63075]: debug 2023-07-29T17:24:34.347+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:34.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[63075]: debug 2023-07-29T17:24:34.347+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:35.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[58650]: debug 2023-07-29T17:24:34.795+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:52.114014+0000 front 2023-07-29T17:23:52.113633+0000 (oldest deadline 2023-07-29T17:24:14.413571+0000) 2023-07-29T17:24:35.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[58650]: debug 2023-07-29T17:24:34.795+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.915287+0000 front 2023-07-29T17:23:57.915501+0000 (oldest deadline 2023-07-29T17:24:23.214921+0000) 2023-07-29T17:24:35.178 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:34 smithi129 bash[58650]: debug 2023-07-29T17:24:34.795+0000 7f7b1eb1b700 -1 osd.5 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.516530+0000 front 2023-07-29T17:24:08.516559+0000 (oldest deadline 2023-07-29T17:24:32.016225+0000) 2023-07-29T17:24:35.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:35 smithi129 bash[60866]: debug 2023-07-29T17:24:35.155+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:35.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:35 smithi129 bash[60866]: debug 2023-07-29T17:24:35.155+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:35.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:35 smithi129 bash[63075]: debug 2023-07-29T17:24:35.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:35.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:35 smithi129 bash[63075]: debug 2023-07-29T17:24:35.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:35.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:35 smithi129 bash[63075]: debug 2023-07-29T17:24:35.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:36.428 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[60866]: debug 2023-07-29T17:24:36.195+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:36.429 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[60866]: debug 2023-07-29T17:24:36.195+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:36.429 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[60866]: debug 2023-07-29T17:24:36.195+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:36.429 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[63075]: debug 2023-07-29T17:24:36.319+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:36.430 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[63075]: debug 2023-07-29T17:24:36.319+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:36.430 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[63075]: debug 2023-07-29T17:24:36.319+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:37.237 INFO:journalctl@ceph.osd.5.smithi129.stdout:Jul 29 17:24:36 smithi129 bash[76044]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-5 2023-07-29T17:24:37.371 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:37 smithi129 bash[60866]: debug 2023-07-29T17:24:37.207+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:37.372 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:37 smithi129 bash[60866]: debug 2023-07-29T17:24:37.207+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:37.372 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:37 smithi129 bash[60866]: debug 2023-07-29T17:24:37.207+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:37.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:37 smithi129 bash[63075]: debug 2023-07-29T17:24:37.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:37.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:37 smithi129 bash[63075]: debug 2023-07-29T17:24:37.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:37.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:37 smithi129 bash[63075]: debug 2023-07-29T17:24:37.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:38.661 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:38 smithi129 bash[60866]: debug 2023-07-29T17:24:38.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:38.661 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:38 smithi129 bash[60866]: debug 2023-07-29T17:24:38.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:38.661 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:38 smithi129 bash[60866]: debug 2023-07-29T17:24:38.231+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:38.662 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:38 smithi129 bash[63075]: debug 2023-07-29T17:24:38.347+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:38.662 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:38 smithi129 bash[63075]: debug 2023-07-29T17:24:38.347+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:38.662 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:38 smithi129 bash[63075]: debug 2023-07-29T17:24:38.347+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:39.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.5.service' 2023-07-29T17:24:39.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:39.409 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-29T17:24:39.410 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-29T17:24:39.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.6 2023-07-29T17:24:39.591 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[63075]: debug 2023-07-29T17:24:39.311+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:39.591 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[63075]: debug 2023-07-29T17:24:39.311+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:39.592 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[63075]: debug 2023-07-29T17:24:39.311+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:39.592 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[60866]: debug 2023-07-29T17:24:39.191+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:39.592 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[60866]: debug 2023-07-29T17:24:39.191+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:39.593 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[60866]: debug 2023-07-29T17:24:39.191+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:39.593 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 systemd[1]: Stopping Ceph osd.6 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:39.928 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[60866]: debug 2023-07-29T17:24:39.563+0000 7f95aa476700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:24:39.928 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[60866]: debug 2023-07-29T17:24:39.563+0000 7f95aa476700 -1 osd.6 125 *** Got signal Terminated *** 2023-07-29T17:24:39.929 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:39 smithi129 bash[60866]: debug 2023-07-29T17:24:39.563+0000 7f95aa476700 -1 osd.6 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:40.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[60866]: debug 2023-07-29T17:24:40.227+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:40.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[60866]: debug 2023-07-29T17:24:40.227+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:40.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[60866]: debug 2023-07-29T17:24:40.227+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:40.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[63075]: debug 2023-07-29T17:24:40.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:40.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[63075]: debug 2023-07-29T17:24:40.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:40.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[63075]: debug 2023-07-29T17:24:40.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:40.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:40 smithi129 bash[63075]: debug 2023-07-29T17:24:40.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:41.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[60866]: debug 2023-07-29T17:24:41.259+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:41.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[60866]: debug 2023-07-29T17:24:41.259+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:41.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[60866]: debug 2023-07-29T17:24:41.259+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:41.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[60866]: debug 2023-07-29T17:24:41.259+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:15.531802+0000 front 2023-07-29T17:24:15.531762+0000 (oldest deadline 2023-07-29T17:24:40.831546+0000) 2023-07-29T17:24:41.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[63075]: debug 2023-07-29T17:24:41.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:41.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[63075]: debug 2023-07-29T17:24:41.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:41.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[63075]: debug 2023-07-29T17:24:41.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:41.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:41 smithi129 bash[63075]: debug 2023-07-29T17:24:41.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:42.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[60866]: debug 2023-07-29T17:24:42.215+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:42.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[60866]: debug 2023-07-29T17:24:42.215+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:42.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[60866]: debug 2023-07-29T17:24:42.215+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:42.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[60866]: debug 2023-07-29T17:24:42.215+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:15.531802+0000 front 2023-07-29T17:24:15.531762+0000 (oldest deadline 2023-07-29T17:24:40.831546+0000) 2023-07-29T17:24:42.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[63075]: debug 2023-07-29T17:24:42.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:42.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[63075]: debug 2023-07-29T17:24:42.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:42.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[63075]: debug 2023-07-29T17:24:42.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:42.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:42 smithi129 bash[63075]: debug 2023-07-29T17:24:42.331+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:43.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[60866]: debug 2023-07-29T17:24:43.171+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:43.678 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[60866]: debug 2023-07-29T17:24:43.171+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:43.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[60866]: debug 2023-07-29T17:24:43.171+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:43.679 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[60866]: debug 2023-07-29T17:24:43.171+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:15.531802+0000 front 2023-07-29T17:24:15.531762+0000 (oldest deadline 2023-07-29T17:24:40.831546+0000) 2023-07-29T17:24:43.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[63075]: debug 2023-07-29T17:24:43.339+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:43.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[63075]: debug 2023-07-29T17:24:43.339+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:43.681 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[63075]: debug 2023-07-29T17:24:43.339+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:43.681 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:43 smithi129 bash[63075]: debug 2023-07-29T17:24:43.339+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:43.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:43 smithi078 bash[78981]: ts=2023-07-29T17:24:43.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:43.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:43 smithi078 bash[78981]: ts=2023-07-29T17:24:43.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:43.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:43 smithi078 bash[78981]: ts=2023-07-29T17:24:43.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:44.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[63075]: debug 2023-07-29T17:24:44.375+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:44.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[63075]: debug 2023-07-29T17:24:44.375+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:44.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[63075]: debug 2023-07-29T17:24:44.375+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:44.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[63075]: debug 2023-07-29T17:24:44.375+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:44.680 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[60866]: debug 2023-07-29T17:24:44.211+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.525322+0000 front 2023-07-29T17:23:50.525433+0000 (oldest deadline 2023-07-29T17:24:16.425532+0000) 2023-07-29T17:24:44.680 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[60866]: debug 2023-07-29T17:24:44.211+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:59.228514+0000 front 2023-07-29T17:23:59.228171+0000 (oldest deadline 2023-07-29T17:24:23.928114+0000) 2023-07-29T17:24:44.680 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[60866]: debug 2023-07-29T17:24:44.211+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:09.631073+0000 front 2023-07-29T17:24:09.630884+0000 (oldest deadline 2023-07-29T17:24:35.530850+0000) 2023-07-29T17:24:44.681 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[60866]: debug 2023-07-29T17:24:44.211+0000 7f95a5269700 -1 osd.6 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:15.531802+0000 front 2023-07-29T17:24:15.531762+0000 (oldest deadline 2023-07-29T17:24:40.831546+0000) 2023-07-29T17:24:44.681 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:44 smithi129 bash[60866]: debug 2023-07-29T17:24:44.211+0000 7f95a5269700 -1 osd.6 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34701.0:865 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94690791036416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) 2023-07-29T17:24:45.787 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:45 smithi129 bash[63075]: debug 2023-07-29T17:24:45.407+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:45.788 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:45 smithi129 bash[63075]: debug 2023-07-29T17:24:45.407+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:45.788 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:45 smithi129 bash[63075]: debug 2023-07-29T17:24:45.407+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:45.789 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:45 smithi129 bash[63075]: debug 2023-07-29T17:24:45.407+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:46.178 INFO:journalctl@ceph.osd.6.smithi129.stdout:Jul 29 17:24:45 smithi129 bash[76280]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-6 2023-07-29T17:24:46.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:46 smithi129 bash[63075]: debug 2023-07-29T17:24:46.359+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:46.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:46 smithi129 bash[63075]: debug 2023-07-29T17:24:46.359+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:46.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:46 smithi129 bash[63075]: debug 2023-07-29T17:24:46.359+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:46.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:46 smithi129 bash[63075]: debug 2023-07-29T17:24:46.359+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:47.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:47 smithi129 bash[63075]: debug 2023-07-29T17:24:47.387+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:47.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:47 smithi129 bash[63075]: debug 2023-07-29T17:24:47.387+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:47.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:47 smithi129 bash[63075]: debug 2023-07-29T17:24:47.387+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:47.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:47 smithi129 bash[63075]: debug 2023-07-29T17:24:47.387+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:48.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.6.service' 2023-07-29T17:24:48.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:48.268 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-29T17:24:48.269 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-29T17:24:48.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.7 2023-07-29T17:24:48.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 systemd[1]: Stopping Ceph osd.7 for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:48.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.351+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:48.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.351+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:48.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.351+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:48.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.351+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:48.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.423+0000 7fb0635c8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-29T17:24:48.681 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.423+0000 7fb0635c8700 -1 osd.7 125 *** Got signal Terminated *** 2023-07-29T17:24:48.681 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:48 smithi129 bash[63075]: debug 2023-07-29T17:24:48.423+0000 7fb0635c8700 -1 osd.7 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T17:24:49.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:49 smithi129 bash[63075]: debug 2023-07-29T17:24:49.371+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:49.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:49 smithi129 bash[63075]: debug 2023-07-29T17:24:49.371+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:49.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:49 smithi129 bash[63075]: debug 2023-07-29T17:24:49.371+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:49.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:49 smithi129 bash[63075]: debug 2023-07-29T17:24:49.371+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:49.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:49 smithi129 bash[63075]: debug 2023-07-29T17:24:49.371+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2023-07-29T17:24:25.132462+0000 front 2023-07-29T17:24:25.132586+0000 (oldest deadline 2023-07-29T17:24:49.232365+0000) 2023-07-29T17:24:50.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:50 smithi129 bash[63075]: debug 2023-07-29T17:24:50.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:50.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:50 smithi129 bash[63075]: debug 2023-07-29T17:24:50.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:50.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:50 smithi129 bash[63075]: debug 2023-07-29T17:24:50.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:50.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:50 smithi129 bash[63075]: debug 2023-07-29T17:24:50.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:50.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:50 smithi129 bash[63075]: debug 2023-07-29T17:24:50.323+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2023-07-29T17:24:25.132462+0000 front 2023-07-29T17:24:25.132586+0000 (oldest deadline 2023-07-29T17:24:49.232365+0000) 2023-07-29T17:24:51.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:51 smithi129 bash[63075]: debug 2023-07-29T17:24:51.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:51.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:51 smithi129 bash[63075]: debug 2023-07-29T17:24:51.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:51.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:51 smithi129 bash[63075]: debug 2023-07-29T17:24:51.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:51.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:51 smithi129 bash[63075]: debug 2023-07-29T17:24:51.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:51.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:51 smithi129 bash[63075]: debug 2023-07-29T17:24:51.307+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2023-07-29T17:24:25.132462+0000 front 2023-07-29T17:24:25.132586+0000 (oldest deadline 2023-07-29T17:24:49.232365+0000) 2023-07-29T17:24:52.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:52 smithi129 bash[63075]: debug 2023-07-29T17:24:52.299+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:52.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:52 smithi129 bash[63075]: debug 2023-07-29T17:24:52.299+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:52.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:52 smithi129 bash[63075]: debug 2023-07-29T17:24:52.299+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:52.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:52 smithi129 bash[63075]: debug 2023-07-29T17:24:52.299+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:52.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:52 smithi129 bash[63075]: debug 2023-07-29T17:24:52.299+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2023-07-29T17:24:25.132462+0000 front 2023-07-29T17:24:25.132586+0000 (oldest deadline 2023-07-29T17:24:49.232365+0000) 2023-07-29T17:24:53.678 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:53 smithi129 bash[63075]: debug 2023-07-29T17:24:53.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-29T17:23:50.923370+0000 front 2023-07-29T17:23:50.923315+0000 (oldest deadline 2023-07-29T17:24:16.823357+0000) 2023-07-29T17:24:53.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:53 smithi129 bash[63075]: debug 2023-07-29T17:24:53.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6814 osd.1 since back 2023-07-29T17:23:57.925158+0000 front 2023-07-29T17:23:57.924976+0000 (oldest deadline 2023-07-29T17:24:23.224966+0000) 2023-07-29T17:24:53.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:53 smithi129 bash[63075]: debug 2023-07-29T17:24:53.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6822 osd.2 since back 2023-07-29T17:24:08.928318+0000 front 2023-07-29T17:24:08.928144+0000 (oldest deadline 2023-07-29T17:24:33.028122+0000) 2023-07-29T17:24:53.679 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:53 smithi129 bash[63075]: debug 2023-07-29T17:24:53.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.78:6830 osd.3 since back 2023-07-29T17:24:16.931364+0000 front 2023-07-29T17:24:16.931255+0000 (oldest deadline 2023-07-29T17:24:39.831162+0000) 2023-07-29T17:24:53.680 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:53 smithi129 bash[63075]: debug 2023-07-29T17:24:53.343+0000 7fb05bbb6700 -1 osd.7 125 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2023-07-29T17:24:25.132462+0000 front 2023-07-29T17:24:25.132586+0000 (oldest deadline 2023-07-29T17:24:49.232365+0000) 2023-07-29T17:24:53.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:53 smithi078 bash[78981]: ts=2023-07-29T17:24:53.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=14 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:53.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:53 smithi078 bash[78981]: ts=2023-07-29T17:24:53.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:24:53.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:24:53 smithi078 bash[78981]: ts=2023-07-29T17:24:53.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:24:55.178 INFO:journalctl@ceph.osd.7.smithi129.stdout:Jul 29 17:24:54 smithi129 bash[76519]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-osd-7 2023-07-29T17:24:56.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@osd.7.service' 2023-07-29T17:24:56.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:56.973 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-29T17:24:56.973 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2023-07-29T17:24:56.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a 2023-07-29T17:24:57.186 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 systemd[1]: Stopping Ceph prometheus.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:24:57.678 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:24:57.679 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2023-07-29T17:24:57.679 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2023-07-29T17:24:57.679 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2023-07-29T17:24:57.679 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2023-07-29T17:24:57.680 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2023-07-29T17:24:57.680 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2023-07-29T17:24:57.680 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.162Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2023-07-29T17:24:57.680 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.163Z caller=main.go:907 level=info msg="Scrape manager stopped" 2023-07-29T17:24:57.680 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.164Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2023-07-29T17:24:57.681 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.164Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2023-07-29T17:24:57.681 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Jul 29 17:24:57 smithi129 bash[71906]: ts=2023-07-29T17:24:57.164Z caller=main.go:1147 level=info msg="See you next time!" 2023-07-29T17:24:57.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@prometheus.a.service' 2023-07-29T17:24:57.978 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T17:24:57.979 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2023-07-29T17:24:57.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c --force --keep-logs 2023-07-29T17:24:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2023-07-29T17:24:58.141 INFO:teuthology.orchestra.run.smithi078.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2023-07-29T17:24:58.141 INFO:teuthology.orchestra.run.smithi078.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2023-07-29T17:24:58.141 INFO:teuthology.orchestra.run.smithi078.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2023-07-29T17:24:58.141 INFO:teuthology.orchestra.run.smithi078.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2023-07-29T17:24:58.141 INFO:teuthology.orchestra.run.smithi078.stderr: ... 2023-07-29T17:24:58.142 INFO:teuthology.orchestra.run.smithi078.stderr:cephadm: error: unrecognized arguments: --keep-logs 2023-07-29T17:24:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-29T17:24:58.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c --force --keep-logs 2023-07-29T17:24:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2023-07-29T17:24:58.314 INFO:teuthology.orchestra.run.smithi129.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2023-07-29T17:24:58.314 INFO:teuthology.orchestra.run.smithi129.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2023-07-29T17:24:58.314 INFO:teuthology.orchestra.run.smithi129.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2023-07-29T17:24:58.315 INFO:teuthology.orchestra.run.smithi129.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2023-07-29T17:24:58.315 INFO:teuthology.orchestra.run.smithi129.stderr: ... 2023-07-29T17:24:58.315 INFO:teuthology.orchestra.run.smithi129.stderr:cephadm: error: unrecognized arguments: --keep-logs 2023-07-29T17:24:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-29T17:24:58.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T17:24:58.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T17:24:58.359 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-29T17:24:58.361 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/remote/smithi078/crash 2023-07-29T17:24:58.362 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/crash -- . 2023-07-29T17:24:58.405 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/remote/smithi129/crash 2023-07-29T17:24:58.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/crash -- . 2023-07-29T17:24:58.422 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-29T17:24:58.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-07-29T17:24:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:grep: /var/log/ceph/eb1107b2-2e30-11ee-9b36-001a4aab830c/ceph.log: No such file or directory 2023-07-29T17:24:58.442 INFO:tasks.cephadm:Compressing logs... 2023-07-29T17:24:58.442 DEBUG:teuthology.orchestra.run.smithi078:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T17:24:58.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T17:24:58.459 INFO:teuthology.orchestra.run.smithi078.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-07-29T17:24:58.480 INFO:teuthology.orchestra.run.smithi129.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-07-29T17:25:03.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:03 smithi078 bash[78981]: ts=2023-07-29T17:25:03.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=14 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:25:03.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:03 smithi078 bash[78981]: ts=2023-07-29T17:25:03.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2023-07-29T17:25:03.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:03 smithi078 bash[78981]: ts=2023-07-29T17:25:03.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2023-07-29T17:25:05.693 INFO:tasks.cephadm:Archiving logs... 2023-07-29T17:25:05.693 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/remote/smithi078/log 2023-07-29T17:25:05.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T17:25:06.479 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/remote/smithi129/log 2023-07-29T17:25:06.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T17:25:07.151 INFO:tasks.cephadm:Removing cluster... 2023-07-29T17:25:07.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c --force 2023-07-29T17:25:11.633 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:25:11 smithi078 systemd[1]: Stopping Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:25:12.781 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:25:12 smithi078 bash[84339]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-node-exporter-a 2023-07-29T17:25:12.781 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:25:12 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:25:12.782 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:25:12 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-29T17:25:12.782 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Jul 29 17:25:12 smithi078 systemd[1]: Stopped Ceph node-exporter.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:25:13.066 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:12 smithi078 systemd[1]: Stopping Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:25:13.067 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:12 smithi078 bash[78981]: ts=2023-07-29T17:25:12.923Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2023-07-29T17:25:13.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:13 smithi078 bash[84478]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-alertmanager-a 2023-07-29T17:25:13.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:13 smithi078 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-29T17:25:13.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Jul 29 17:25:13 smithi078 systemd[1]: Stopped Ceph alertmanager.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:25:36.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c --force 2023-07-29T17:25:41.335 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:25:40 smithi129 systemd[1]: Stopping Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:25:42.169 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:25:41 smithi129 bash[77089]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-node-exporter-b 2023-07-29T17:25:42.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:25:41 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:25:42.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:25:41 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-29T17:25:42.170 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Jul 29 17:25:41 smithi129 systemd[1]: Stopped Ceph node-exporter.b for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:25:42.429 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:25:42 smithi129 systemd[1]: Stopping Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c... 2023-07-29T17:25:43.361 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:25:43 smithi129 bash[77230]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c-grafana-a 2023-07-29T17:25:43.362 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:25:43 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@grafana.a.service: Main process exited, code=exited, status=143/n/a 2023-07-29T17:25:43.362 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:25:43 smithi129 systemd[1]: ceph-eb1107b2-2e30-11ee-9b36-001a4aab830c@grafana.a.service: Failed with result 'exit-code'. 2023-07-29T17:25:43.362 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Jul 29 17:25:43 smithi129 systemd[1]: Stopped Ceph grafana.a for eb1107b2-2e30-11ee-9b36-001a4aab830c. 2023-07-29T17:26:17.999 INFO:tasks.cephadm:Removing cephadm ... 2023-07-29T17:26:18.000 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-29T17:26:18.007 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-29T17:26:18.013 INFO:tasks.cephadm:Teardown complete 2023-07-29T17:26:18.013 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-29T17:26:18.022 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-29T17:26:18.022 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T17:26:18.025 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T17:26:18.045 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2023-07-29T17:26:18.046 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2023-07-29T17:26:18.046 INFO:teuthology.orchestra.run.smithi078.stdout:*hv01.front.sepi 44.4.53.2 2 u 67 64 377 0.110 1.270 0.599 2023-07-29T17:26:18.046 INFO:teuthology.orchestra.run.smithi078.stdout:+hv02.front.sepi 63.231.80.2 3 u 65 64 377 0.090 6.618 0.488 2023-07-29T17:26:18.046 INFO:teuthology.orchestra.run.smithi078.stdout:+hv03.front.sepi 74.6.168.72 3 u 53 64 377 0.099 1.371 0.643 2023-07-29T17:26:18.046 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-07-29T17:26:18.073 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2023-07-29T17:26:18.074 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2023-07-29T17:26:18.074 INFO:teuthology.orchestra.run.smithi129.stdout:*hv01.front.sepi 44.4.53.2 2 u 57 64 377 0.083 -0.965 0.079 2023-07-29T17:26:18.074 INFO:teuthology.orchestra.run.smithi129.stdout:+hv02.front.sepi 63.231.80.2 3 u 3 64 377 0.082 4.346 0.073 2023-07-29T17:26:18.074 INFO:teuthology.orchestra.run.smithi129.stdout:+hv03.front.sepi 74.6.168.72 3 u 62 64 377 0.103 -1.884 0.085 2023-07-29T17:26:18.074 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-07-29T17:26:18.075 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-29T17:26:18.085 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-29T17:26:18.086 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-29T17:26:18.195 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-29T17:26:18.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-29T17:26:18.297 INFO:teuthology.task.internal:Duration was 2115.203556 seconds 2023-07-29T17:26:18.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-29T17:26:18.326 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-29T17:26:18.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T17:26:18.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T17:26:18.387 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-29T17:26:18.388 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2023-07-29T17:26:18.389 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T17:26:18.402 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2023-07-29T17:26:18.402 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T17:26:18.420 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-29T17:26:18.421 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T17:26:18.447 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T17:26:18.525 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-29T17:26:18.526 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T17:26:18.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T17:26:18.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-29T17:26:18.689 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-29T17:26:18.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T17:26:18.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T17:26:18.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-29T17:26:18.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T17:26:18.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T17:26:18.759 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2023-07-29T17:26:18.776 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2023-07-29T17:26:18.796 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T17:26:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T17:26:18.803 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T17:26:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T17:26:18.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-29T17:26:18.819 INFO:teuthology.task.internal:Transferring archived files... 2023-07-29T17:26:18.820 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/remote/smithi078 2023-07-29T17:26:18.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T17:26:18.854 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902/remote/smithi129 2023-07-29T17:26:18.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T17:26:18.882 INFO:teuthology.task.internal:Removing archive directory... 2023-07-29T17:26:18.883 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T17:26:18.894 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T17:26:18.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-29T17:26:18.911 INFO:teuthology.task.internal:Not uploading archives. 2023-07-29T17:26:18.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-29T17:26:18.957 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-29T17:26:18.958 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T17:26:18.960 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T17:26:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 17:26 /home/ubuntu/cephtest 2023-07-29T17:26:18.969 INFO:teuthology.orchestra.run.smithi129.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 17:26 /home/ubuntu/cephtest 2023-07-29T17:26:18.971 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-29T17:26:19.123 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-29T17:26:19.194 INFO:teuthology.nuke:Checking targets against current locks 2023-07-29T17:26:19.226 DEBUG:teuthology.nuke:shortname: smithi078 2023-07-29T17:26:19.227 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T17:26:19.256 DEBUG:teuthology.nuke:shortname: smithi129 2023-07-29T17:26:19.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T17:26:19.261 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 16:42:02.580587', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-07-29T17:26:19.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:03:42-orch-reef-release-distro-default-smithi/7356902', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 16:42:02.582212', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-07-29T17:26:19.325 INFO:teuthology.orchestra.console.smithi078:Power off 2023-07-29T17:26:19.326 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T17:26:19.346 INFO:teuthology.orchestra.console.smithi129:Power off 2023-07-29T17:26:19.347 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T17:26:19.359 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2023-07-29T17:26:19.360 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:19.376 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2023-07-29T17:26:19.377 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:19.391 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-07-29T17:26:19.412 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-07-29T17:26:23.392 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:23.414 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:23.506 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-07-29T17:26:23.528 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-07-29T17:26:27.508 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:27.530 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:27.622 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-07-29T17:26:27.641 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-07-29T17:26:31.628 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:31.643 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T17:26:31.741 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2023-07-29T17:26:31.759 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2023-07-29T17:26:31.842 INFO:teuthology.orchestra.console.smithi078:Power off completed 2023-07-29T17:26:31.861 INFO:teuthology.orchestra.console.smithi129:Power off completed 2023-07-29T17:26:31.970 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} duration: 2115.2035562992096 failure_reason: 'Command failed on smithi078 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb1107b2-2e30-11ee-9b36-001a4aab830c -- bash -c ''ceph orch upgrade ls''"' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f0613ce4061c4998a7abb954b5cd5691 status: fail success: false 2023-07-29T17:26:31.971 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-29T17:26:32.062 INFO:teuthology.run:FAIL